AI Recurrent Neural Networks

Why Order Matters

In many types of data, the order of elements carries meaning: 'the dog bit the man' and 'the man bit the dog' contain the exact same words but mean very different things, and a stock price 30 days from now depends on the order of price movements that led up to it, not just an unordered bag of numbers. A plain feedforward network (including a CNN) expects a fixed-size input and has no built-in concept of 'what came before this point' - which makes it a poor fit for sequences of varying length where earlier context changes the meaning of later elements.

A Hidden State That Remembers

A recurrent neural network (RNN) processes a sequence one element at a time - one word, one time step, one audio sample - rather than all at once. At every step, it combines the current input with a hidden state: a set of numbers summarizing what the network has seen so far. The network updates this hidden state after each step and carries it forward to the next one, so information from earlier in the sequence can keep influencing how later elements are interpreted.

# A simplified sketch of how a hidden state carries information forward.
# Real RNNs learn the update rule; here we just illustrate the idea with
# a running "memory" that never fully resets between words.

words = ["the", "weather", "is", "sunny", "today"]
hidden_state = 0  # starts empty, will accumulate a simple signal

for word in words:
    word_signal = len(word)  # a stand-in for a real learned representation
    hidden_state = 0.5 * hidden_state + word_signal
    print(f"After '{word}': hidden_state = {hidden_state:.2f}")

The Challenge of Long Sequences

Plain RNNs struggle to hold on to information from many steps earlier. As the hidden state gets updated again and again across a long sequence, the influence of something said at step 2 tends to fade out by step 50 - a version of the vanishing gradient problem, this time playing out across time steps instead of layers. This makes plain RNNs unreliable at connecting information that is far apart in a long sequence.

LSTMs and GRUs: Built-In Gates for Better Memory

To address this, researchers designed improved recurrent units - most notably the LSTM (Long Short-Term Memory) and the simpler GRU (Gated Recurrent Unit). Both add small internal gates that learn what to keep, what to forget, and what to pass through at every step, giving the network a much longer effective memory than a plain RNN without needing to work through the gate math by hand - it's enough to know that these gates were specifically designed to fight the fading-memory problem.

• Predicting the next word while typing or generating text
• Machine translation, where meaning depends on word order in both languages
• Speech recognition, converting a sequence of audio samples into words
• Time-series forecasting, such as predicting demand or sensor readings from a history of past values