Recurrent Neural Networks (RNNs)

Standard ANNs and CNNs have no concept of "Time". They look at a piece of data entirely statically. If you feed the word "Apple" then the word "Tree", the network does not natively remember that it saw "Apple" previously.

Recurrent Neural Networks (RNNs) solve this by introducing loops into the architecture, allowing information to persist. They are designed for sequential data like Time-Series predicting (stock prices) or Natural Language Processing (translating English to French word-by-word).

1. The RNN Architecture

Instead of passing data strictly straightforwardly from Input -> Hidden -> Output, an RNN's hidden layer feeds back into itself. At Time Step 2 (processing the second word in a sentence), the hidden layer receives two inputs simultaneously: 1. The actual data at Time Step 2. 2. The hidden layer's own state from Time Step 1 (the Context/Memory).

2. The Vanishing Gradient Problem

Standard RNNs suffer mathematically on long sequences. Because the Backpropagation algorithm calculates gradients by continually multiplying numbers between 0 and 1, the gradient exponentially shrinks to nearly zero. The network physically cannot connect a word at the beginning of a long paragraph to a word at the end of the paragraph.

3. Long Short-Term Memory (LSTM) Networks

LSTMs are an advanced form of RNN designed explicitly to solve the Vanishing Gradient problem. They introduce complex "Gates" inside the neuron. - Forget Gate: Decides what information from the past is no longer relevant and should be deleted from memory (e.g., if a new sentence starts with a new subject). - Input Gate: Decides what new information from the current time step is important enough to be added to long-term memory. - Output Gate: Decides what the cell outputs to the next layer based on its newly updated memory state.

Before Transformers dominated NLP, LSTMs powered Google Translate, Siri voice recognition, and massive algorithmic trading systems.

How to execute the examples:

Go to the Examples/ folder and run the script: python RNN_TimeSeries.py