Essentials of Deep Learning: A Practical Introduction to LSTMs

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

An LSTM (Long Short-Term Memory) is a recurrent neural network that carries a learned state through a sequence. Its gates regulate what information to retain, update, and expose, helping address the vanishing-gradient problem that can make ordinary RNNs struggle with long-range dependencies. LSTMs remain useful for moderate-sized, ordered data and streaming workloads, but they are not automatically the best choice: compare them with simple baselines, GRUs, temporal CNNs, or Transformers for the task at hand.

What is sequence data?

Sequence data consists of observations whose order matters. A reading may mean something different depending on what came before it, so treating every observation as an independent row can discard useful context.

#	Preview	Product	Price
1		Python Deep Learning: Understand how deep neural networks work and apply them to real-world tasks	$39.99	Buy on Amazon
2		Childrens Learn to Read Books Lot 60 - First Grade Set + Reading Strategies NEW Buyer's Choice	$28.50	Buy on Amazon
3		Understanding Deep Learning: Building Machine Learning Systems with PyTorch and TensorFlow: From...	$54.34	Buy on Amazon
4		Deep Learning with Keras: Implementing deep learning models and neural networks with the power of...	$47.64	Buy on Amazon
5		Deep Learning for Beginners: A beginner's guide to getting up and running with deep learning from...	$21.39	Buy on Amazon

• Time series: measurements such as temperature, demand, or sensor output.
• Text: words, subword units, or characters in order.
• Audio: successive frames or signal segments.
• Events: ordered user actions, machine logs, or transaction records.

Recurrent neural networks process such inputs step by step, passing a state forward from one time step to the next. TensorFlow describes RNNs as a fit for sequence data including time series and natural language: TensorFlow’s RNN guide.

Why feed-forward networks and vanilla RNNs can struggle

A standard feed-forward network maps an input to an output without an inherent mechanism for carrying information from one sequence position to the next. An RNN adds a recurrent state, but that state is repeatedly transformed as the sequence is processed. During training through time, gradients can become very small (vanish) or very large (explode), making it difficult to learn relationships between events far apart in the sequence.

#1 Best Overall

Sale

Python Deep Learning: Understand how deep neural networks work and apply them to real-world tasks

$39.99

Buy on Amazon

The original LSTM work addressed the difficulty of learning over extended time intervals when error signals decay during recurrent backpropagation. It was published by Sepp Hochreiter and Jürgen Schmidhuber in 1997: the paper in Neural Computation and its PubMed record.

An LSTM is designed to improve information and gradient flow; it does not eliminate optimization problems, guarantee long-term recall, or make longer input windows useful by themselves.

What an LSTM keeps: two related states

At time step t, an LSTM maintains a cell state, c_t, and a hidden state, h_t. The cell state provides a comparatively direct memory pathway through time. The hidden state is the exposed output used by later layers and passed into the next step. They are related, but they are not interchangeable.

x_t ──► [LSTM cell] ──► h_t
          ▲       │
       h_(t-1)  c_t
          ▲       │
       c_(t-1) ◄──┘

The diagram is schematic: the recurrent computation uses both prior states and the current input to calculate the new states.

How the gates update the cell

Gates are learned numerical transformations, not symbolic rules or conscious decisions. Sigmoid outputs lie between zero and one, scaling how much of a component passes through. The candidate update uses a hyperbolic tangent to produce proposed new content.

Forget gate

The forget gate scales the previous cell state. A value near one retains the corresponding component; a value near zero attenuates it.

Rank #2

Sale

Childrens Learn to Read Books Lot 60 - First Grade Set + Reading Strategies NEW Buyer's Choice

• Childrens Learn to Read Books Lot 60 - First Grade Set + Reading Strategies NEW
• 60 stapled booklets total. 15 titles each in levels A, B, C, and D
• Each 8-page reader is black and white as designed by a reading specialist to attract attention to the print
• Measures 4 1/2" by 5 1/2"
• This series of books is a Teachers' Choice award winning item as voted by Learning Magazine!

$28.50

Buy on Amazon

Input gate and candidate update

The input gate scales a candidate update derived from the current input and previous hidden state. Together, they control how much new content is added to the cell state.

Cell state and output gate

The cell state combines the scaled previous state with the scaled candidate. The output gate then controls how much of the updated cell state is exposed as the hidden output.

i_t = σ(W_ii x_t + b_ii + W_hi h_(t-1) + b_hi)
f_t = σ(W_if x_t + b_if + W_hf h_(t-1) + b_hf)
g_t = tanh(W_ig x_t + b_ig + W_hg h_(t-1) + b_hg)
o_t = σ(W_io x_t + b_io + W_ho h_(t-1) + b_ho)
c_t = f_t ⊙ c_(t-1) + i_t ⊙ g_t
h_t = o_t ⊙ tanh(c_t)

Here x_t is the current input; i, f, and o are the input, forget, and output gates; g is the candidate update; σ is the sigmoid function; and ⊙ means elementwise multiplication. Equivalent equations and implementation details appear in the PyTorch LSTM documentation.

The additive cell-state update creates a comparatively direct path for information and gradients, unlike a simple RNN’s repeated nonlinear state transformation. In a temperature series, a model might retain a slow trend while updating short-term variation. That is an intuition, not a guarantee that particular units acquire human-readable meanings. What it retains depends on learned parameters, model size, data, and training.

Choose the output shape for the task

In Keras, the standard input shape is (batch, timesteps, features). For example, (32, 24, 19) represents 32 sequences, each containing 24 steps and 19 features. A univariate window of length 24 is commonly represented as (samples, 24, 1). Text token IDs normally pass through an embedding layer before an LSTM.

The Keras layer API documents input shape, state options, masking, and optimized execution conditions: TensorFlow Keras LSTM API.

Rank #3

Sale

Understanding Deep Learning: Building Machine Learning Systems with PyTorch and TensorFlow: From Neural Networks (CNN, DNN, GNN, RNN, ANN, LSTM, GAN) to Natural Language Processing (NLP)

$54.34

Buy on Amazon

• return_sequences=False (the default) returns the final output, often suitable for classifying or regressing an entire window.
• return_sequences=True returns an output at every time step. Use it for token-level labeling, per-step prediction, or as input to another recurrent layer.
• return_state=True also returns the final hidden and cell states.

TensorFlow’s time-series tutorial illustrates the distinction in output shapes: time-series tutorial.

Variable-length inputs require consistent padding and masking. TensorFlow documents conditions for its optimized GPU path, including right-padding when masking is used and particular activation and dropout settings; performance is not automatic across every configuration.

Prepare time-series windows without leakage

For forecasting, preserve chronology throughout data preparation and evaluation. A frequent error is to form overlapping windows across the full series and then randomly divide those windows, allowing near-duplicate periods or future information to contaminate the training evaluation.

1. Sort observations by time and define training, validation, and test periods in chronological order.
2. Fit any scaler or imputer on the training period only; apply those fitted transformations to later periods.
3. Create windows within the appropriate partitions. For a window covering steps t−23 through t, a one-step-ahead target is generally the value at t+1, unless the task explicitly predicts the current step.
4. Keep feature ordering and preprocessing identical at training and inference.
5. Batch examples as (samples, timesteps, features) and verify target shapes against the intended horizon.
6. Evaluate on a later holdout period. For changing time series, use rolling or walk-forward validation as well.

Before tuning a neural model, compare it with persistence (the last value), a moving average, and suitable linear, seasonal, or tree-based approaches. If the LSTM does not beat a relevant baseline on a realistic holdout, added complexity may not be justified.

Build a minimal forecasting model with Keras

The example below expects prepared floating-point arrays: X_train shaped (samples, 24, 19) and y_train containing the correctly aligned next-step targets. Create validation arrays from a later period using the same training-fitted preprocessing.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Input(shape=(24, 19)),
    layers.LSTM(64),
    layers.Dense(1)
])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="mse",
    metrics=[keras.metrics.MeanAbsoluteError()]
)

model.fit(
    X_train, y_train,
    validation_data=(X_valid, y_valid),
    epochs=50,
    callbacks=[keras.callbacks.EarlyStopping(
        monitor="val_loss", patience=5, restore_best_weights=True
    )]
)

test_predictions = model.predict(X_test)

Epoch count and patience here are example settings, not universal recommendations. Select them using validation performance and compare test results only after choices are settled. For predictions in original units, invert any target scaling fitted on the training data. A point prediction and an MAE or MSE score do not describe forecast uncertainty; high-stakes forecasting may require quantile, interval, ensemble, or probabilistic methods.

Rank #4

Sale

Deep Learning with Keras: Implementing deep learning models and neural networks with the power of Python

$47.64

Buy on Amazon

Equivalent PyTorch model

With batch_first=True, PyTorch accepts input shaped (batch, sequence, features). Its LSTM returns a sequence output plus final hidden and cell states. This model uses the final sequence output for a one-value prediction:

import torch
from torch import nn

class SequenceModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            batch_first=True
        )
        self.output = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        sequence_output, (hidden, cell) = self.lstm(x)
        return self.output(sequence_output[:, -1, :])

The last-step extraction fits a many-to-one output. For per-step predictions, apply the output layer to the full sequence_output. PyTorch’s API also documents multilayer and bidirectional configurations, dropout, projection options, and state shapes: nn.LSTM documentation.

Use an LSTM for text generation

A character-level generator predicts the next character from preceding characters; word- and subword-level models use different tokenization and vocabulary trade-offs. Training examples pair prefixes with next-token targets. With teacher forcing, training supplies the true preceding tokens; generation is different: the model feeds each sampled token back into its next input, one step at a time.

1. Normalize and tokenize the training text, and reserve later or held-out material for evaluation rather than fitting vocabulary decisions on test content.
2. Build prefix/next-token examples and an embedding-plus-LSTM model with a vocabulary-sized softmax output.
3. Train with a next-token classification loss, commonly categorical cross-entropy for one-hot targets or sparse categorical cross-entropy for integer token IDs.
4. At generation time, sample from the predicted distribution. Temperature adjusts distribution sharpness; top-k or top-p sampling restricts candidate tokens.
5. Inspect outputs for repetition, incoherence, and passages reproduced from training data; compare against held-out next-token performance, not just fluent-looking samples.

Small educational generators may memorize examples or fall into repetitive loops. Their output is not evidence of general language understanding. The historical tutorial’s text-generation example is useful as a concept, but its older framework framing should not be treated as current setup guidance: Analytics Vidhya’s tutorial.

When an LSTM is a good choice—and when it is not

LSTMs can suit sequence classification, sensor and event data, moderate-sized forecasting problems, and streaming inference where state is updated as observations arrive. Character-level generation is also a useful learning exercise. These are possible applications, not guarantees of accuracy.

• Try an LSTM when order carries information, sequence lengths are moderate, a compact recurrent model meets latency limits, or streaming state is useful.
• Prefer a simpler method when lag features, seasonal baselines, linear models, classical forecasting, or boosted trees perform similarly—or when the problem is mostly tabular.
• Consider a GRU when you want a gated recurrent alternative with a simpler state design. It may perform similarly or differently; measure rather than assume.
• Consider a temporal CNN when a bounded receptive field and parallel computation are attractive.
• Consider a Transformer when long-range interactions, parallel training, or pretrained attention-based models matter and the available compute supports them. Attention-based models are central to many language workloads, but they do not universally win every sequence task; see TensorFlow’s Transformer tutorial.

Also account for deployment: recurrent processing is sequential across time, and stateful inference requires careful state handling. A bidirectional LSTM uses both directions within its input sequence, so it is unsuitable for causal real-time forecasting if the backward direction would use observations unavailable at prediction time. TensorFlow lists SimpleRNN, GRU, and LSTM among its recurrent layers: Keras RNN guide.

Best Value

Sale

Deep Learning for Beginners: A beginner's guide to getting up and running with deep learning from scratch using Python

$21.39

Buy on Amazon

Common errors and how to diagnose them

Future information leaks into training

Look for scalers fitted to the full dataset, random splits of time-dependent observations, overlapping windows that cross the split boundary, target-derived features, or bidirectional processing that can see future values. Rebuild partitions chronologically and fit transformations using training data only.

Inputs and targets do not line up

Check a few window indices by hand. If the task is one step ahead, the target must occur after the input window. A model can achieve an impressive score while learning the wrong alignment if features accidentally contain the target.

Output shape does not match the task

For a stacked LSTM, an intermediate recurrent layer generally needs return_sequences=True. For sequence labeling, returning only the final output discards the intermediate representations needed for per-step predictions. Verify a batch’s input, output, and target shapes before a long training run.

State crosses a boundary unexpectedly

stateful=True carries state between batches. It requires deliberate batch ordering and explicit resets at sequence or partition boundaries; do not enable it merely because the data is sequential.

Training improves but later-period results worsen

A falling training loss paired with rising validation loss suggests overfitting. Try fewer units, dropout or weight decay, early stopping, or better representative data. For unstable training, gradient clipping, a lower learning rate, or shorter windows may help. LSTMs can still have exploding gradients.

Historical performance fails after a regime change

Time series can change distribution. Use later-period and walk-forward evaluation, monitor performance after deployment, and treat historical accuracy as evidence about the tested periods—not a guarantee about future ones.

Finance results look stronger than they are

An LSTM applied to historical prices does not establish a profitable trading strategy. A credible financial evaluation must address look-ahead and survivorship bias, transaction costs, slippage, and regime changes, using a realistic out-of-sample design.

Is LSTM still relevant?

Yes, for the right workload. LSTMs remain a practical option for compact recurrent models, streaming state, and moderate-scale sequential problems. For large language workloads or problems dominated by long-range relationships, investigate attention-based models; for small forecasting tasks, first establish whether a simpler method is enough. Choose with a temporally valid benchmark, an appropriate baseline, and the actual latency, memory, and data constraints of deployment—not an assumption that one architecture is best for every sequence.

Quick Recap

SaleBestseller No. 1

Python Deep Learning: Understand how deep neural networks work and apply them to real-world tasks

$39.99

SaleBestseller No. 2

Childrens Learn to Read Books Lot 60 - First Grade Set + Reading Strategies NEW Buyer's Choice

Childrens Learn to Read Books Lot 60 - First Grade Set + Reading Strategies NEW; 60 stapled booklets total. 15 titles each in levels A, B, C, and D

$28.50

SaleBestseller No. 3

Understanding Deep Learning: Building Machine Learning Systems with PyTorch and TensorFlow: From Neural Networks (CNN, DNN, GNN, RNN, ANN, LSTM, GAN) to Natural Language Processing (NLP)

$54.34

SaleBestseller No. 4

Deep Learning with Keras: Implementing deep learning models and neural networks with the power of Python

$47.64

SaleBestseller No. 5

Deep Learning for Beginners: A beginner's guide to getting up and running with deep learning from scratch using Python

$21.39

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.