Time Series Analysis Part 4 (Deep Learning)

We mentioned in the previous section that in the last few years, there have been many attempts at a fresh approach to statistical time-series forecasting using methods like neural networks and Deep Learning.

Machine Learning and Deep Learning are subsets of Artificial intelligence, as discussed here. Deep learning excels at identifying patterns in unstructured data, which most people know as media such as images, sound, video and text.

Consequently, the usage of neural networks for time series data comes as a natural evolution. Scientific research hints that forward feeding artificial neural networks tend to outperform ARIMA models, but this is not always the case in real life, thus more research is required.

A type of neural networks particularly suited for time series analysis is the Recurrent Networks and particular the Long Short-Term Memory Units (LSTMs).

Recurrent nets are a type of artificial neural network designed to recognize patterns in sequences of data, such as text, genomes, handwriting, the spoken word, or numerical times series data. They are arguably the most powerful type of neural network, for data that can be treated as a sequence.

Recurrent networks possess a certain type of memory, and memory is also part of the human condition.

Humans don’t start their thinking from scratch every second. As we read, for example, we understand each word based on your understanding of previous words. We don’t throw everything away and start thinking from scratch again. Our thoughts have persistence (or at least for most of us…).

Traditional machine learning and neural networks can’t do this, and it seems like a major shortcoming. For example, imagine that we want to classify what kind of event is happening at every point in a movie. It’s unclear how a traditional neural network could use its reasoning about previous events in the film to inform later ones.

Recurrent neural networks address this issue. They are networks with loops in them, allowing information to persist.

Long Short-Term Memory (LSTM) is a type of recurrent neural network that can learn the order dependence between items in a sequence, therefore have the promise of being able to learn the context required to make predictions in time series forecasting problems, rather than having this context pre-specified and fixed. Research suggests that Long Short-Term Memory (LSTM) is able to solve many time series tasks unsolvable by feed-forward networks using fixed size time windows, but has limitations (for example it is not very suitable for AR-based univariate time series forecasting).

LSTMs were a big step in what we can accomplish with Deep Learning. It’s natural to wonder: is there another big step? A common opinion among researchers is: “Yes! There is a next step and it’s attention!” The idea is to let every step of an RNN pick information to look at from some larger collection of information. For example, if you are using an RNN to create a caption describing an image, it might pick a part of the image to look at for every word it outputs. In fact, Xu, et al.(2015) do exactly this – it might be a fun starting point if you want to explore attention! There’s been a number of really exciting results using attention, and it seems like a lot more is around the corner…

Attention isn’t the only exciting thread in RNN research. For example, Grid LSTMs by Kalchbrenner, et al. (2015) seem extremely promising. Work using RNNs in generative models – such as Gregor, et al. (2015)Chung, et al. (2015), or Bayer & Osendorfer (2015) – also seems very interesting. The last few years have been an exciting time for recurrent neural networks, and the coming ones promise to only be more so!

This Blog is created and maintained by Iraklis Mardiris

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s