A recurrent neural network (RNN) processes an ordered sequence one step at a time, combining each new input with a hidden state carried from the previous step. That state is a learned, compact summary of earlier context, so past inputs can influence later outputs. Vanilla RNNs establish this basic idea; LSTM and GRU cells add gates to make long-range information easier to preserve.
What is a recurrent neural network?
An RNN is a neural-network family designed for data in which order matters. Text, speech, sensor readings, financial time series and event logs are not just collections of independent records: changing the order can change the meaning. An RNN handles that dependence by reusing the same recurrent transition at every position in a sequence.
As an Amazon Associate I earn from qualifying purchases.
At time step t, the cell receives the current input vector x_t and the previous hidden state h_(t-1). It computes a new state:
h_t = f(x_t, h_(t-1))
An output can then be calculated from h_t, for example by applying a linear layer and a task-specific activation. The function f and all of its parameters are shared across time steps. A sequence of any supported length can therefore be processed with one set of learned weights rather than a separate network for every position.
#1 Best Overall
The hidden state is best understood as a learned summary of prior context, not a perfect transcript of every earlier input. During training, the model learns which aspects of that summary help predict the next token, classify a sequence, estimate a future value or produce another output.
How does an RNN remember previous inputs?
Step-by-step state updates
- Read the first item. The model combines
x_1with an initial state, often a vector of zeros or a learned initial vector, to produceh_1. - Carry the state forward. At the next position it combines
x_2withh_1, producingh_2. The same transition and weights are reused. - Repeat through the sequence. Each
h_tcontains information selected from the current input and the preceding state, so context can flow forward. - Read an output. A model may emit an output at every step, use the final state for one sequence-level prediction, or pass all states to another network.
For language modeling, for example, the state after the words already seen is used to estimate the next word. For sequence classification, the state or a pooled collection of states can feed a classifier. For speech or sensor streams, outputs can be generated continuously as observations arrive.
Unrolling the recurrence
Although an RNN is one reusable cell, it is often drawn “unrolled” as a chain of copies, one per time step. The copies share parameters but receive different inputs and states. This view makes both the computation and the training process clear: information moves forward through the chain, while error signals move backward through it.
Why do RNNs have a vanishing-gradient problem?
RNNs are trained with backpropagation through time. The training algorithm unrolls the recurrence over a sequence and propagates the loss gradient through every transition that separates an earlier input from a later prediction. Those repeated derivatives involve products of matrices and activation-function derivatives.
If the factors in that product tend to have magnitudes below one, the gradient can shrink toward zero as it travels farther back. This is the vanishing-gradient problem: updates for early time steps become so small that the model struggles to learn dependencies spanning many positions. If the factors are too large, the gradient can instead grow explosively, producing unstable updates or numerical overflow. Stanford’s recurrent-network teaching materials identify vanishing gradients as a central difficulty in recurrent models.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
The problem is not that a vanilla RNN is unable to carry any information. It can learn short-range patterns effectively, and it remains a useful conceptual and experimental baseline. The difficulty is preserving a useful learning signal over long gaps, especially when a later output depends on an event that occurred many steps earlier.
Vanilla RNN, LSTM and GRU: what is the difference?
“RNN” can mean the broad family of recurrent networks or the simplest ungated cell often called a vanilla RNN. LSTM and GRU are specific gated recurrent-cell designs within that family.
Recommended Free Tools
| Variant | State design | Strengths | Trade-offs and fit |
|---|---|---|---|
| Vanilla RNN | One hidden state updated directly from the current input and previous hidden state. | Small conceptual footprint; useful as a teaching and baseline model. | Long-range learning can be difficult because gradients may vanish or explode. |
| LSTM | A hidden state plus a separate cell state, controlled by learned gates. | Gates regulate what to retain, add and expose, helping error signals travel across longer intervals. | More state and parameters than a simple cell; compute and memory requirements increase. |
| GRU | A hidden state controlled by reset and update gates. | Gated information flow with a different, generally more compact design than an LSTM. | Exact equations depend on the implementation; task-specific validation is still required. |
How an LSTM controls information flow
An LSTM maintains a cell state in addition to its exposed hidden state. Learned gates regulate three broad operations: deciding what old information to retain, deciding what new information to add, and deciding what part of the cell state to expose as the hidden state. This creates a comparatively direct path for information and error signals instead of forcing every update through one unconstrained nonlinear transformation.
Hochreiter and Schmidhuber introduced LSTM to address what they called “insufficient, decaying error backflow.” In their 1997 paper, they reported that, on artificial data, the proposed method learned minimal time lags in excess of 1,000 discrete-time steps. That result is an experimental finding under the paper’s conditions, not a guarantee that every LSTM application can remember arbitrary-length histories.
How a GRU updates its state
A GRU uses reset and update gates rather than a separate exposed cell state. In the documented PyTorch implementation, the gates combine the current input and prior hidden state to form a candidate state, then blend that candidate with the old hidden state. The update gate controls how much of the previous state survives; the reset gate controls how strongly prior state contributes when forming the candidate.
Rank #3
PyTorch notes that its new-gate computation differs subtly from the equations in the original GRU paper and from some other frameworks. When reproducing a paper or moving weights between libraries, use the target framework’s documentation and verify the exact gate equations, parameter order and bias handling.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
When should you use an LSTM or GRU?
There is no source-supported universal winner among vanilla RNN, LSTM and GRU. Choose experimentally, using the requirements of your sequence and validation results on your own data.
- Start with a vanilla RNN when you need a transparent baseline, your dependencies are short, or you are teaching and debugging the recurrence itself.
- Try an LSTM when retaining information over longer gaps is important and the additional cell state and gates fit your compute and memory budget.
- Try a GRU when you want gated recurrence with a different state design and need to compare its accuracy, latency and parameter count against an LSTM.
- Measure rather than assume. Compare validation loss or task metrics, training stability, inference latency, memory use and parameter count under the same preprocessing and sequence lengths.
Sequence length, available latency, batch size, hardware, parameter budget and whether predictions must be causal can change the best choice. A cell that performs well on one data set or implementation is not automatically superior on another.
Bidirectional and stacked RNNs
Bidirectional recurrence
A bidirectional RNN runs one recurrent network from the beginning of a sequence and another from the end, then combines their states. Each position can therefore use left and right context. This is useful when the complete sequence is available, such as labeling an already-recorded sentence.
It is unsuitable for strictly real-time prediction when future inputs are not known: the backward direction would require information that has not arrived yet. For streaming inference, use a causal, forward-only architecture or a deliberately limited look-ahead window.
Rank #4
Stacked or deep recurrence
A stacked RNN places recurrent layers on top of one another. The first layer can represent local temporal patterns, while higher layers receive its sequence of states and can build more abstract features. Depth increases representational capacity, but also adds computation, memory use and optimization complexity. Compare a deeper model with a simpler one using the same validation protocol and an explicit latency budget.
What is an RNN encoder-decoder?
In an encoder-decoder, one recurrent network reads an input sequence and converts it into a representation; a second recurrent network uses that representation to generate a target sequence. The encoder and decoder may have separate parameters and can be designed with different input and output vocabularies or feature dimensions.
Cho and colleagues introduced this formulation for statistical machine translation. Their paper reported that phrase-pair conditional probabilities from the recurrent encoder-decoder improved an existing log-linear translation model when added as a feature. That is a historical, paper-specific result, not a claim that encoder-decoder RNNs currently provide the best quality for modern machine translation.
A basic encoder-decoder compresses the source into a fixed-length representation, which can become a bottleneck for long inputs. Later recurrent systems added mechanisms such as attention, and other neural architectures are also used today. The core encoder-decoder idea remains useful for understanding how one sequence can be mapped to another.
How an RNN training workflow works
1. Represent the sequence
Convert each time step into a numeric vector. Text may use token IDs mapped to embeddings; sensor data may use normalized measurements; audio systems may use frames or learned features. Preserve the order and record sequence lengths when examples have different durations.
Best Value
2. Choose the output arrangement
- Many-to-one: consume a sequence and make one prediction, such as document or activity classification.
- Many-to-many, aligned: emit one output per input step, such as tagging or frame-level labeling.
- Many-to-many, generated: encode one sequence and decode another, as in the encoder-decoder formulation.
- One-to-many: start from a vector or initial state and generate a sequence.
3. Decide how state is handled
For independent examples, reset the hidden state between sequences. For a continuous stream split into chunks, you may carry state across chunks, but detach it from the previous computation graph so memory use does not grow without bound. Make the boundary policy explicit: accidentally carrying state between unrelated examples leaks information and corrupts training.
4. Train through time
Unroll each training sequence or chunk, compute the loss, and backpropagate through the unrolled steps. Long sequences increase memory use and can intensify unstable gradients. Gradient clipping is a practical safeguard when exploding gradients cause erratic updates, but it does not solve a vanishing-gradient design problem by itself.
5. Validate the deployment conditions
Evaluate on sequences whose lengths, missing-data patterns and timing resemble production inputs. Test both ordinary cases and long-range cases that require information from earlier in the sequence. For a streaming system, measure end-to-end latency with the state update performed one step at a time; a bidirectional model’s offline accuracy is not a valid comparison for a causal service.
Free tools Windows power users keep installed
One-click scans. No signup required.
Common mistakes and fixes
- Calling every recurrent cell “an LSTM.” State clearly whether the model is vanilla RNN, LSTM or GRU; the terms describe different cell mechanisms.
- Assuming the hidden state stores everything. It is a learned summary with finite capacity. Check performance as the dependency gap grows.
- Ignoring gradient instability. Monitor loss and gradient norms, use suitable initialization and learning rates, and consider clipping for exploding gradients.
- Using future context accidentally. Do not use a bidirectional model or random time shuffling when the deployed predictor must operate causally.
- Porting GRU weights without checking equations. Gate order and new-gate placement can differ between PyTorch and other implementations; follow the destination framework’s specification.
- Comparing cells with unequal budgets. Report parameter counts, sequence lengths, training settings and latency so an apparent accuracy advantage is interpretable.
Further reading
For a broader mathematical treatment, the Deep Learning textbook by Ian Goodfellow, Yoshua Bengio and Aaron Courville includes Chapter 10, “Sequence Modeling: Recurrent and Recursive Nets.” The book site presents it as a resource for students and practitioners. The original 1997 LSTM paper is the primary source for the long-time-lag experiment and its motivation; the encoder-decoder paper by Cho and colleagues provides the historical sequence-to-sequence formulation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
