Driver FixRecommendedSound, Wi-Fi or graphics acting up? Check drivers firstFind missing or outdated drivers fast.Check DriversHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsSlow PC?RecommendedPC slow today? Run a repair scan before it gets worseResolve common Windows issues and optimize system performance.Scan Now×
FREEDOM251

Freedom251 › Blog › code search

From a Test-Suite Trace to a Search Signal: the Bootstrap Pipeline Story

By Freedom251 Team · · 6 min read

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The most useful map of a codebase’s live logic, according to this case study, comes from recording which source functions each test actually executes, not from searching the text. In a 1,727-test Python suite, a custom tracer linked 1,551 tests (89.8%) to at least one source function. That execution-based map became the edge source for the whole pipeline, while name matching and file-level matching were judged too loose to serve that role. Every figure below comes from Mikhail’s September 22, 2026 DEV Community post, a first-person engineering account. The benchmarks are the author’s own, and the post does not describe independent reproduction.

The question behind the pipeline

The author starts from a blunt question: where is the real business logic here, and what can I safely throw away? Ordinary code search answers a narrower one. It finds where a string or symbol appears, but it cannot say which functions take part in live execution. A name that appears in many files can look central while the path that actually runs stays hidden behind generic names.

As an Amazon Associate I earn from qualifying purchases.

Four stages, one hard problem

The proposed bootstrap runs four stages in order:

  1. Entities. Types and data classes that describe what the system stores and passes around.
  2. Entry points. Externally reachable code, including functions decorated with @mcp_app.tool.
  3. Tests as execution evidence. Each test is linked to the source functions it runs. This is the technically difficult stage.
  4. Git history. Commits are mined for architectural decision records.

The third stage is where most of the engineering effort goes. The bootstrap must connect a test to the exact functions it executes, and a single test rarely maps to a single function.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Why name matching and file imports fell short

The obvious first attempt was to link tests to code by name. None of the 109 sampled tests named the function they executed, so name-based links were not usable.

Matching at file level, by following imports, reached 77.9%. The author rejected it anyway, because a file can hold many functions and the link points to the wrong one too often to act as a target.

A Tarantula ranking heuristic, a fault-localization technique that scores code by how strongly it correlates with failing tests, fared better but still fell short. It supplied a rank-at-most-three candidate for 22.6% of tests and a rank-one candidate for 7.5%. The author rejected it as a universal primary-target selector and kept the full trace when building TESTS edges.

Linking tests to functions with a trace

The custom Python plugin is built on sys.settrace. Across the 1,727-test suite it covered 1,212 unique source functions. Each linked test touched 10.1 source functions on average, with a median of 6 and a range from 1 to 118.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

That spread is the central lesson. One test commonly exercises many functions, so a model that assumes one test maps to one function would misrepresent the suite. The edges have to be many-to-many.

Mocks are the main exception to watch. The author reports that about 10.2% of tests (176 of 1,727) executed no source function at all. Static companion analysis covered 88 of those 176.

How static analysis compares with the trace

The author compared three static methods against the dynamic trace as the reference: AST direct calls (L1), name tokens (L2), and file imports (L3). The results, as reported by the author in the September 22, 2026 post:

Method Hit Recall Precision Mean candidates per test
L1, AST direct calls 88.4% 30.3% 68.0% 2.9
L2, name tokens 17.7% 3.8% 12.1% not stated
L3, file imports 91.6% 72.0% 21.8% 41.4
Union of L1, L2, and L3 90.4% 70.0% 20.6% not stated

The pattern explains the author’s conclusion. File imports recover most of what the trace finds, but they carry 41.4 candidates per test on average and only 21.8% precision. Direct calls are the most precise method, yet they recover under a third of the trace-observed links. The author therefore treats static analysis as a useful companion and candidate supplier for mock tests, while the dynamic trace remains the edge driver.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What tracing costs

Overhead is the price of that precision, and the post reports two measurements:

Comparison Baseline Traced run Reported overhead Context
Custom sys.settrace plugin 174.8 s 198.6 s 13.6% Same-session comparison
coverage run with Python 3.14 sys.monitoring 184.88 s 221.78 s 19.96% Separate comparison; the author describes it as about 1.5 times slower than the custom plugin

Because edges must be refreshed to stay current, the trace cost recurs every time the suite is re-run for that purpose, not once.

Wiring test links into search

In the experiment the post labels E17, the graph held 1,727 tests, 1,595 Test nodes, 16,172 TESTS edges, and 1,132 covered functions. Search reaches these links through SymbolIndexAdapter.get_tests_for_symbol(), and Searcher._append_tests_signal() appends the results. The integration is bounded in several ways:

  • Up to three tests are attached per function.
  • The post describes the per-query cap as min(len, 6).
  • Tests receive a graph_score of 0.4, while definitions receive 1.0, so tests read as context rather than competing primary hits.
  • The signal sits behind the MSCODEBASE_TESTS_SIGNAL toggle, which was off by default in the implementation described.

The cost on the graph stage was measurable. In the wider panel, the average graph-stage time rose from 6.52 ms to 7.53 ms when the signal was added. The post reports this as a 15.3% increase. The rounded millisecond values work out to about 15.5%, so treat the percentage as approximate. The figure comes from one panel and is not a production latency forecast.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What the search panels showed

A seven-function A/B panel kept function ranking unchanged, with MRR of 1.000 in both arms. Six of the seven queries received relevant covering tests.

The wider 35-query panel gave the following results:

Measure Signal off Signal on
hit@1 33/35 (94.3%) 33/35 (94.3%)
hit@3 34/35 (97.1%) 34/35 (97.1%)
MRR 0.957 0.957
Responses with covering tests added not applicable (no signal) 34/35 (97.1%)

The author states what this does and does not show: “TESTS-signal does not improve hit@1 (off=on). It only adds context (tests) to already-found results.” (Mikhail, DEV Community, September 22, 2026.) In practice, the signal changes what a reader sees beside a result, not which result ranks first.

Where the method stops: language coverage

The dynamic edge builder is Python-first. TESTS edges exist for 1,108 of 3,256 Python functions (34.0%). The Go/Rust group (716 functions) and the TypeScript group (11 functions) produced none. The Go and TypeScript connectors are proposed in the post but not implemented in the work described.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The author also ran three smaller external checks, which are useful as illustrations and should not be read as a portability benchmark:

  • gemma_agent (Python): 2,882 tests, 2,874 passing; 97.3% linked (2,805 tests); overhead of 17.4% (71.4 s against 60.8 s).
  • commit- (Python): 27 tests; 100% linked.
  • codebase-memory-mcp (Go): 27 test functions; coverage figures of 51.0% (package) and 22.2% (per test).
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Known risks in the trace and the search path

  • Failing tests remove edges, so a red suite under-reports what is covered.
  • Mocked tests can execute no source functions (about 10.2% of tests in this suite).
  • Widely used utility functions create noisy links to many unrelated tests.
  • Some test nodes carry line number 0.
  • Graph reindexing can shift node order.
  • Very large suites may not fit within CI time windows.
  • The graph_score constant has not been tested against BM25 or reranker interactions.
  • Verification was local. Clean CI confirmation was pending a pull request merge when the post was written.
  • The 35-query panel used one primary codebase and no deep reranker interaction.

What you can safely discard, and what you cannot

The post stops short of a deletion rule. The following is a cautious reading of its findings, based on the author’s own caveats:

  1. Get the suite green first. Because failing tests remove edges, code exercised only by failing tests will look unused.
  2. Treat a missing link as unverified, not dead. Mocks, Python-only coverage, and the test-failure gap all produce silence that does not prove absence. In Go, Rust, or TypeScript code, this method says nothing at all.
  3. Check entry points and entities before trusting the trace. Code reachable from an externally exposed entry point is live behavior even when no test reaches it.
  4. Do not prune shared helpers by their link count. Widely used utilities attract many links and look important, but they are often plumbing rather than business logic.
  5. Use static candidates to suggest, not to delete. Direct calls and file imports are useful for proposing where to look, but the post’s own precision figures show they cannot justify removal.

Trade-offs to weigh before adopting it

Compare any alternative on the same six axes the author uses:

  • Precision against recall, and how many candidates a reviewer must check per target.
  • Execution overhead every time the edges are refreshed.
  • Language and test-framework coverage, where the Python-first builder is the binding constraint.
  • Whether the output improves ranking or merely adds context, as the panels show.
  • Suite and CI reliability, including suites long enough to strain CI windows.
  • Breadth of validation, including downstream LLM use, which the post leaves as future work.

The author’s own conclusion is that the trace drives the edges and the search signal adds context rather than better top-one ranking. For a Python codebase with a reliable, reasonably fast suite, the approach offers a practical way to separate exercised code from the rest. Outside Python, or with a flaky or slow suite, the evidence in the post does not support the same use.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

code searchLLM retrievalPythonsoftware engineeringStatic analysisTest Coverage

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Outdated Drivers Are Slowing You DownFree scan - exact matches
Windows Errors? Fix Them Before They SpreadFree repair scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.