Use token-first search when developers need to find exact identifiers, paths, error strings, or literals. Use embedding-based retrieval when they describe behavior in natural language that uses different words from the code. If your repository search must handle both kinds of query, test a hybrid system—but choose from measured results on your own codebase, not a presumed universal winner.
How token-first and embedding search retrieve code
Token-first search matches words and symbols
Lexical search represents documents through their terms and scores matches using methods such as TF-IDF or BM25. It is strongest when the query vocabulary overlaps with the indexed code or its surrounding text: a function name, an error message, a path, or a configuration literal. Google Cloud describes sparse token-based methods as not usually encoding semantic meaning on their own, so a query phrased as an idea may miss code that expresses the same idea in different words. Google Cloud’s overview of hybrid search explains the distinction.
As an Amazon Associate I earn from qualifying purchases.
Embeddings retrieve by learned similarity
Embedding systems convert text or code into vectors and retrieve items that are nearby in that learned space. The potential benefit for code search is bridging a vocabulary gap: a developer can describe what a function does without knowing its name or the terms its author used. The trade-off is that semantic similarity is not exact matching. Results may be conceptually related while omitting the exact symbol or line the developer needs.
This gap is a recognized code-search problem, not a guarantee that any particular embedding model will solve it. The 2019 CodeSearchNet paper framed semantic code search as matching natural-language queries to relevant code whose vocabulary can differ from the query. Its dataset contained about 6 million functions across Go, Java, JavaScript, PHP, Python, and Ruby, alongside about 2 million automatically generated query-like descriptions obtained by scraping and preprocessing function documentation. Those are dataset scale figures, not a head-to-head finding that embeddings beat lexical search. Read the CodeSearchNet paper.
#1 Best Overall
Which approach fits which developer query?
| Query or need | Likely useful starting point | Why |
|---|---|---|
| Exact function, class, or variable name | Token-first | The target’s identifier is the query; lexical matching makes the match visible and direct. |
| Error text, literal, acronym, or file path | Token-first | These queries depend on exact terms that semantic similarity may blur. |
| Natural-language description using the code’s own vocabulary | Either; establish a lexical baseline | Lexical search may already find the relevant terms, so embeddings should demonstrate added value rather than be assumed necessary. |
| Natural-language description using different words from the code | Embeddings, or hybrid | Vector similarity may connect the intent to code written with different terminology. |
| Workload containing both exact and intent-based queries | Evaluate hybrid retrieval | It can combine lexical and semantic result signals, but adds system and tuning complexity. |
These are starting hypotheses, not substitutes for repository-specific evaluation. Query quality, tokenization, the searchable fields, code chunking, filters, and ranking all affect the outcome.
What hybrid retrieval adds—and what it does not
Hybrid retrieval combines lexical and vector searches, then merges or reranks their results. Google Cloud, Elastic, and Microsoft document hybrid approaches; Microsoft describes merging BM25 and vector result lists with Reciprocal Rank Fusion (RRF). Microsoft’s hybrid search overview and Elastic’s semantic-text hybrid workflow describe examples of these patterns. They establish that hybrid is a practical architecture option, not that it wins for every repository or query mix.
Consider hybrid when exact-symbol searches matter alongside descriptions of behavior that use different vocabulary. Evaluate whether it improves useful results at the depth your developers or coding agent actually consume. If it does not improve that workload enough to justify maintaining both retrieval paths and their fusion settings, the simpler system may be the better choice.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Design the code index around useful context
Keep code-aware boundaries
Embedding a codebase is not just a matter of sending arbitrary text windows to a model. The Qdrant Team’s code-search cookbook recommends candidate chunks aligned with language structures such as functions, methods, structs, and enums, balancing meaningful context against model input limits. It also describes enriching chunks with comments, docstrings, and metadata. Its demonstration uses separate models for natural-language and code-to-code similarity and combines natural-language function-signature results with implementation snippets; that is an implementation example, not a universal model or chunking prescription. See the Qdrant Team cookbook.
Rank #3
Treat filtering and presentation as part of retrieval
Search quality depends on what happens after ranking, too. GitLab’s implemented semantic code-search design describes optional directory restriction, configurable neighbor and result counts, filtering sensitive or excluded files, grouping results by path, merging overlapping line ranges, and calculating an overall confidence level from result scores. These are product-specific design details, but they illustrate why retrieval should be judged on the context a user receives—not just the vector or lexical score. The design page is marked implemented and dated 2026-06-29; its defaults and API details may change. Read GitLab’s semantic code-search design.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Evaluate retrieval on your own repository
- Build a representative query set. Include exact function and class names, errors, paths, acronyms, behavior descriptions, and descriptions that deliberately use different words from the implementation.
- Label relevant code regions. Decide what counts as useful for each query: the exact declaration, a caller, an implementation, or a broader context region. Keep those judgments consistent.
- Compare baselines fairly. Run lexical and embedding retrieval against the same corpus snapshot, chunking, filters, and result depth. Record whether the exact target appears and inspect false positives as well as misses.
- Measure at the usable cutoff. Report relevance at the number of results a developer or downstream agent can actually inspect. A ranking that improves deep in the list may not help a workflow that only consumes the first few results.
- Test hybrid where the query mix warrants it. Compare its fused list against both baselines, especially on exact-token and vocabulary-gap cases, and decide whether additional coverage justifies the added complexity.
- Check freshness and operations. Make a small edit, rename or move a symbol, then measure when the index reflects it. Also compare indexing and refresh behavior, latency, privacy constraints, and operating cost in the deployment you would use.
- Keep query-level diagnostics. Use misses and noisy results to identify whether the problem lies in chunking, analyzers, embeddings, filters, or fusion settings.
The cited sources do not establish a neutral, universal latency, freshness, cost, or relevance winner for codebase context retrieval. Those outcomes depend on the repository, implementation, and deployment, so measure them rather than extrapolating from a vendor’s documented workflow.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute
