Autoregressive models, variational autoencoders (VAEs), normalizing flows, and generative adversarial networks (GANs) make complex data learnable in different ways: they factor probabilities, introduce latent variables, transform densities, or train through an adversarial game. The practical difference is what each model computes during training and generation—and whether it provides an explicit likelihood for an individual example.
What does it mean to learn a complex distribution?
A generative model aims to represent the distribution of observed data, such as images or text, so it can assign plausibility to examples or generate new ones. The central challenge is that the data may have many dimensions and intricate dependencies. Each model family makes that challenge tractable through a different structure and objective.
As an Amazon Associate I earn from qualifying purchases.
For models that explicitly represent likelihood, a common objective is to make the model distribution resemble the data distribution. Minimizing the Kullback–Leibler divergence from the data distribution to the model is equivalent, with respect to model parameters, to minimizing cross-entropy: the data entropy is constant as the model changes. Because the true data distribution is unknown, training estimates the expectation using examples, yielding negative log-likelihood minimization. This explanation applies to likelihood-based approaches, not to the original GAN minimax objective. The DEV Community overview presents likelihood-based and likelihood-free approaches as a useful introductory distinction, not a complete taxonomy of every variant.
How does each family represent the distribution?
| Family | Structural idea | Training and density | Main trade-off |
|---|---|---|---|
| Autoregressive models | Factor the joint distribution into ordered conditional probabilities. | Can optimize likelihood by evaluating conditional probabilities. | Sequential generation can be slow; training parallelism depends on the architecture and factorization. |
| Variational autoencoders | Introduce a latent variable and model observations conditioned on it, with an approximate inference model for latent states given observations. | Optimize a variational lower bound, or ELBO, when exact posterior inference is intractable. | The latent representation and inference approximation shape what is learned. |
| Normalizing flows | Apply invertible transformations to a simple probability density. | Use the transformations to calculate how density changes and evaluate density. | Invertibility constrains the transformations; computational cost varies by design. |
| GANs | Train a generator and discriminator together in a minimax adversarial game. | The original objective uses the discriminator’s learning signal rather than explicit per-example likelihood as its central quantity. | Training depends on the interaction between two models and uses a different objective from likelihood maximization. |
Autoregressive models: factor the joint probability
The chain rule gives an exact factorization of a joint distribution into conditional distributions. For variables ordered as x1, …, xn, the identity is p(x) = ∏i=1n p(xi | x1, …, xi−1). The identity itself is exact; a learned model only approximates the conditionals from data, and the chosen ordering affects practical behavior.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Generation follows that factorization: sample the first variable, then use it to generate the next, continuing in sequence. This dependency can make generation slow when there are many variables. Pixel Recurrent Neural Networks provide a concrete example: they predict image pixels sequentially. The Pixel Recurrent Neural Networks paper describes this sequential pixel prediction. Training parallelism, however, depends on the particular architecture and factorization; sequential generation does not by itself establish that all training computations must also be sequential.
VAEs: infer a latent explanation for an observation
A latent variable, commonly written z, is an unobserved representation used to model how an observation x could be generated. A VAE specifies a distribution p(z) over latent states and a generative distribution p(x | z), often implemented as a decoder. To learn from a given observation, it also uses an inference model—sometimes called an encoder or recognition model—to approximate the posterior distribution over latent states given x.
Rank #2
Exact posterior inference can be intractable. Rather than assume the inference model gives the true posterior, a VAE trains it together with the generative model using the evidence lower bound (ELBO), a variational lower bound on the observation’s log probability. The approximation and objective affect the representation learned; the inference model should not be described as identical to the true posterior. Kingma and Welling’s Auto-Encoding Variational Bayes introduces this approach and motivates the recognition model by the difficulty of exact inference.
Normalizing flows: transform a simple density
A normalizing flow starts with a variable whose probability density is straightforward to calculate, then maps it through a sequence of invertible transformations. Because each transformation can be reversed, the model can relate a point in the transformed space to its starting point and account for how the transformation changes density. This enables explicit density calculations while allowing the sequence of transformations to build a more complex distribution.
Invertibility is both the enabling idea and a design constraint: transformations must preserve enough information to be reversed, so not every possible mapping is available. The resulting computational cost depends on the chosen transformations; invertible architectures do not all have identical costs. Rezende and Mohamed’s paper studies normalizing flows for variational inference, a specific application of the method.
GANs: learn through an adversarial game
A GAN has two models. The generator produces samples, while the discriminator tries to distinguish samples from the data from those produced by the generator. Training places them in a minimax game: the generator seeks to produce samples the discriminator cannot distinguish, while the discriminator learns to tell the sources apart. Goodfellow and coauthors describe their framework as simultaneously training a generative model that captures the data distribution and a discriminative model that estimates whether a sample came from the training data rather than the generator. The original GAN paper gives that formulation.
Rank #4
In the original GAN setup, the discriminator supplies the generator’s learning signal; explicit likelihood for each example is not the central training objective. The discriminator is not simply a direct density estimator. This distinction separates the original adversarial objective from likelihood-based training, without implying that no GAN variant can ever be combined with a likelihood-related method.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Which modeling approach fits the problem?
There is no universal ranking among these families. The useful choice depends on whether the task needs explicit likelihoods, what form of inference or generation is practical, and which training objective suits the application.
Best Value
- Choose an autoregressive formulation when ordered conditional probabilities and likelihood evaluation fit the task, while accounting for sequential generation.
- Choose a VAE formulation when a latent representation and approximate inference are useful, and the ELBO-based objective is appropriate.
- Choose a flow formulation when invertible transformations and tractable density calculations fit the model, accepting the architectural constraints they impose.
- Choose an adversarial formulation when the generator–discriminator game is the desired learning approach and explicit per-example likelihood is not the central objective.
These distinctions explain how the families make complex distributions computationally approachable; they do not establish a head-to-head winner. The cited material does not report a single comparison of all four families on the same dataset and compute budget.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
