Recommended Free Tools
A word embedding is a list of numbers, called a vector, that a model learns for each word so that words used in similar contexts end up in similar positions. The numbers are not dictionary definitions. They are useful because training shapes them so that relationships between words become measurable as distances and directions. This article builds that idea step by step, starting from the problem embeddings solve and ending with the difference between static and contextual versions.
The problem embeddings solve
A machine-learning model cannot work with the word “horse” directly. It needs numbers. The simplest way to turn a vocabulary into numbers is a one-hot code: every word gets its own position in a long vector, a 1 marks that position, and every other position is 0. That representation is exact, but it carries no information about meaning. Under one-hot coding, “horse” and “burro” are as different from each other as “horse” and “orange.” Each pair of distinct words sits the same distance apart.
As an Amazon Associate I earn from qualifying purchases.
Dense embeddings replace that isolated coding with a much shorter vector, often tens or hundreds of numbers, where every number is learned during training. Because the positions are no longer assigned by hand, the model can place related words near each other. That is the core reason embeddings are useful: they can encode relationships that a one-hot code cannot express at all, as Google for Developers’ embeddings course explains when it contrasts the two approaches.
One-hot codes compared with learned vectors
The table below uses a three-word vocabulary. The one-hot values are exact. The dense values are invented for illustration and are not output from any trained model; their only purpose is to show what “closeness” can look like once training has adjusted the numbers.
| Word | One-hot code (3 positions) | Illustrative dense vector (2 numbers, not from a real model) |
|---|---|---|
| horse | [1, 0, 0] | [0.80, 0.10] |
| burro | [0, 1, 0] | [0.70, 0.20] |
| orange | [0, 0, 1] | [-0.30, 0.90] |
| Distance between “horse” and “burro” | Same as any other pair of different words | Small: the two vectors point to nearby positions |
The one-hot column cannot tell the model that two animal words are related. The dense column can, but only because the numbers were adjusted so that it does. The rest of this article explains how that adjustment happens.
How training turns context into geometry
An embedding is learned, not written by hand. The main idea is that a word’s meaning in practice is revealed by the words around it. Training exploits that by repeatedly presenting the model with examples drawn from a large body of text, then adjusting the vectors whenever the model predicts the surrounding context badly.
Word2vec is the classic teaching example of this process. It was introduced in 2013 and remains useful for illustration, although Google for Developers describes it as an older method that is largely superseded by newer approaches. The steps below show the logic of a context-prediction method in general terms.
- Read a corpus. Start with a large collection of text. The vocabulary is built from the words that occur in it, and each word starts with a vector of random numbers.
- Choose a target and its neighbors. Slide a window across the text. In a sentence such as “the burro carried water up the hill” (a hypothetical example), the target might be “burro” and the neighbors “the,” “carried,” and “water.”
- Predict the context. The model uses the target’s current vector to score how likely each nearby word is to appear beside it.
- Adjust the vectors. When the prediction is poor, the numbers are nudged so that the observed context becomes more likely. The adjustment changes the vectors of the words involved.
- Repeat across the corpus. Over millions of such examples, words that keep appearing in similar contexts receive similar vectors.
Why similar contexts produce similar vectors
The learning signal is indirect. Nobody tells the model that horses and burros are both animals that are ridden or led, or that they share a category. The model is only asked to predict what appears nearby. Google’s course uses the pair “burro” and “horse” to illustrate this: when both words show up in similar sentence settings, the training process tends to pull their vectors toward each other. The relationship emerges from the pattern of usage in the text.
What the learned space does and does not mean
Once the vectors exist, they form a space. Comparing two words means comparing their positions, usually by distance or by the angle between vectors. This is a practical tool, but it has limits worth keeping in mind:
- Closeness is a relative measure. It tells you that two words behave similarly in the training text, not that they are synonyms.
- Individual dimensions have no guaranteed label. A single axis does not reliably mean something such as “animalness,” and it is a mistake to read it that way.
- The vectors depend on the corpus and the training setup. Changing the text or the settings changes the numbers, so a set of embeddings is a learned representation for a particular purpose, not a universal dictionary.
- One vector set is not automatically best for every task. The useful test is performance on the application you care about.
Static and contextual embeddings
Word2vec-style methods are static: each word gets one vector regardless of where it appears. That creates a specific limitation. Take “orange.” In one sentence it names a fruit; in another it names a color. A static model places both uses at the same location, because the word has only one entry in its vocabulary.
Contextual embeddings address this by letting the surrounding words affect the representation of each occurrence. The same written word can receive different representations in different sentences, so the fruit sense and the color sense can be separated. Google for Developers’ material on obtaining embeddings describes this distinction and gives examples of contextual methods.
| Property | Static embeddings (for example, word2vec) | Contextual embeddings |
|---|---|---|
| Vectors per written word | One fixed vector per vocabulary entry | Can differ for each occurrence, depending on the sentence |
| Handling of “orange” (fruit or color) | One location for both senses | Different occurrences can be represented differently |
| Main teaching value | Clear picture of how context-prediction shapes a space | Captures sentence-level meaning, at the cost of a more complex model |
| Typical role today | Older, still useful for illustration (per Google for Developers) | The approach that most modern systems use (per the same course material) |
The static model is the right place to start because its mechanics are simpler to follow. Contextual methods build on the same underlying idea of learning from usage, but they add machinery that lets the representation depend on the sentence.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What the 2013 word2vec paper reported
The original word2vec paper, “Efficient Estimation of Word Representations in Vector Space,” was published in 2013 by Tomas Mikolov, Kai Chen, Greg S. Corrado, and Jeffrey Dean. Its abstract reports that the authors could learn high-quality word vectors from a dataset of 1.6 billion words in less than a day. That figure is a result the authors reported for their own setup in 2013. It is a historical data point about the method’s efficiency at that time, not a benchmark for current hardware or current models.
The paper’s abstract opens by stating its purpose: “We propose two novel model architectures for computing continuous vector representations of words from very large data sets.” Read that sentence as a description of the goal, which is compact, learned vectors computed from large amounts of text.
A practical path after the concept
Once the idea of learned vectors is clear, the most productive next step is to see one in use rather than read more definitions. A workable sequence is:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →- Work through the embeddings material in Google for Developers’ machine-learning course, paying attention to the sections on embedding space and static embeddings.
- Open TensorFlow’s “Word embeddings” tutorial, which trains an embedding layer inside a sentiment-classification model and shows how to visualize the result.
- Look at the trained vectors yourself. Check which words end up near each other and ask whether the neighbors make sense for the text you trained on. Vectors learned from a different corpus may give different neighbors, which is the corpus dependence described above.
- Only then move to contextual models, where the same vocabulary item can receive different representations in different sentences.
Working in this order keeps each new idea tied to a concrete object, a vector you can inspect, rather than to a vocabulary of terms.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
