Category: Uni notes

Time series forecasting was the topic of week 10’s lecture. To complete time series forecasting we first need to remove anything that is easy to forecast from the data:

• Trends
• Cycles (Cyclical Components)
• Seasonal variations
• ‘Irregular’ the hardest to predict and the component which our neural networks will be attempting to forecast.

Autocorrelation generally stronger for recent data items and degrades in quality as we step back through the time series. Autocorrelation uses past data items in an attempt to predict n time steps into the future. It must be noted that error incurred in lesser timesteps forward are more than likely to grow as the prediction continues to step forward. Although this point seems obvious when viewing data predictions that seem intuativley correct our own confirmation bias often outweighs the awareness of a models limitations.

Spatio-Temporal models incorporate a principal component. This is a variable/s who’s influence on future timesteps is significant. An example in our water use prediction would be the rainfall of previous months. Low rainfall would suggest higher water usage. Their are many methods for identifying Principal components, Karl Pearson invented this field with the introduction of his Pearson Product-moment analysis.

Forecasting linear time series can be conducted using a Single layer perceptron. It may however be questionable as to how much this tool would be superior as opposed to more simplistic modelling methods. Auto-regressive with external variables [Arx] models utilize both previous time series data and principal component states for generating forecasts.

Evaluating model accuracy can be done in rudamentary fashion using root mean square error [RMSE].

Moving past the simplisting Single layer networks we review time lagged feed forward networks:

We then moved to Non-Linear Auto-regressive with external variable [NArx] networks:

The same training principles as with standard NNs applies to time series forecasting. Importantly the training data must be viewed in chronological order as forcasting would suggest in contrast to classification.

Again, awareness must be given to over/under fitting. Minimizing RMSE on training data does not infer an accurate model for all/future data.

Natural computation’s 9th week saw an introduction to associative memory networks.

Initialization of an Bi-Directional Associative memory network involves establishing a weight matrix using input and output pairs:

It seems much easier to write a simple script which demonstrates understanding of the weight initialization and memory recall algorithms. Hopefully I can do that this week. The major question that comes to mind after this lecture was why these networks would be used instead of a SOM.

With a unit test and assignment this week, there were no new topics introduced. The research I had to do for SOMs in relation to the assignment did yield a lot of new information.

Implementation of SOM networks has a number variable components:

• Neighbour updating, neighbour radius
• Weight decay
• Random weight initialization (the random weights at initialization will effect clusters)
• Adjusting learning rate and learning rate decay
• Adjusting training/test data split
• Adjusting number of neurons in 2D lattice

Evaluation of the quality of clusters created by a SOM is quite difficult. Weight distances is the best method for checking if like clusters have formed in different sections of the map. Running some clustering in MatLab yielded basic results but I am not familiar enough with the clustering tool to extrapolate all of of the information required to make inference on the results.

The information gained from self-organizing maps may be useful when constructing supervised learning networks.