An AI agent earned its cost in this benchmark only where the question needed adaptive investigation. Where the required data was already structured and the query could be expressed as typed choices, a non-generative route matched the agent’s accuracy at a fraction of the latency. In Anant Kumar’s author-reported test over 100 questions and 2,951 Wikipedia articles, the largest accuracy gain came from giving the system structured graph data, not from adding an agent. Treat this as one carefully described case study rather than a general rule for agent return on investment.
What was measured
Kumar built six pipelines over 2,951 Wikipedia articles and scored them on the same 100 questions. The questions covered lookup, temporal, multi-hop, superlative, and aggregation tasks. Every pipeline used the same generation model, Gemini 3.1 Flash-Lite, with local BGE embeddings and TigerGraph’s native vector index. Answers were scored by exact match against gold answers, with no language model in the scoring loop. The benchmark was built for the TigerGraph Agentic GraphRAG Hackathon, and the write-up is published on DEV Community under Kumar’s name. The figures below are his, reported for 2026, and I did not reproduce them independently.
As an Amazon Associate I earn from qualifying purchases.
The numbers
The write-up describes six pipelines, but the results it reports in comparable form cover five configurations. The table lists those five. Blank-looking cells are marked “not stated” where the write-up gives no value.
| Configuration | Exact match | Tokens per question | Latency per question | Generation-model calls |
|---|---|---|---|---|
| RAG (top-five retrieval) | 67% | 3,586 | Not stated | Not stated |
| GraphRAG (entity linking, one-hop traversal) | 67% | 3,952 | Not stated | Not stated |
| Agent over text and entity tools only | 70% | 6,065 | Not stated | Not stated |
| Agent with structured graph tools | 99% | 3,412 | 13.2 seconds | Not stated |
| Typed selection planner with structured graph tools | 99% | 2,267 | 1.7 seconds | Zero, as reported for the planning route |
Read the table in two steps. Moving from RAG to the text-and-entity agent added three percentage points. Moving to the agent with structured graph tools added twenty-nine more. The agent itself was a small part of the reported gain; the data representation and the tool surface it could call were the larger part.
Where the gain came from
Agents over unstructured tools gained little
The text-and-entity agent reached 70%, against 67% for both RAG and GraphRAG. It also used the most tokens of any configuration in the table, at 6,065 per question. In this corpus, an agent that could search text and follow entity links investigated more but did not answer materially more questions correctly.
Structured graph tools closed the gap
The jump to 99% came when the agent could query structured data directly. Kumar parsed fields from Wikipedia infoboxes into an Olympic-event graph, with links to Games, Sport, and Venue and a previous-Games edge. Questions that depend on exact fields and relationships became lookups and filters rather than guesses from retrieved prose.
Aggregation questions exposed retrieval’s limit
Aggregation is where retrieval struggled most. Answering a question such as “how many cycling events had more than 30 competitors?” requires every matching record, not a sample. Top-five retrieval can surface only a handful of candidates, so a count built from it is wrong by construction. The write-up reports 1 of 21 aggregation questions correct for RAG and 0 of 21 for GraphRAG. After the structured graph ingestion described above, it reports 21 of 21.
Free tools Windows power users keep installed
One-click scans. No signup required.
The lesson is about the shape of the question. Counts, rankings across a full set, and filters over complete records need a store that can enumerate them. Narrative questions that a few relevant passages can answer do not.
When planning adds nothing
The full agent scored 99% at 13.2 seconds per question. Replacing its generative planner with two typed selection calls kept the same 99% and brought latency to 1.7 seconds, with zero generation-model calls on the planning route. Kumar’s explanation is that when every routing decision maps to an existing row or a typed option, planning becomes selection. The system chooses from a known set rather than reasoning about an open action space.
He also reports a practical motive. A 500-calls-per-day free-tier limit interrupted benchmark work, which made a route that avoids generation calls more than a speed preference. That detail concerns his own setup; your limits will differ.
A rough screen for a new task:
- If the answer is a fixed set of rows, fields, or categories that you can enumerate, test a typed query path first.
- If answering requires counts, totals, or rankings over the complete set, make sure the store can return all matching records before comparing any model.
- If the question requires following leads across documents, reformulating searches, or deciding what to look up next, an agent with tools is the case the write-up supports most clearly.
- If the routing choices are open-ended text rather than a finite list, expect the generative planner to remain necessary.
Failures that changed the numbers
Three problems in the write-up are worth knowing, because each one moved a score or nearly did.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsFluent answers fooled the judge
An LLM judge rated 14 wrong answers 4 or 5 out of 5, often because the wrong answers were fluent refusals. Kumar therefore leaned on exact match and added a pass that checks whether the answer is supported by the retrieved evidence. For any evaluation, a judge that rewards tone can inflate accuracy.
A field-selection change cut accuracy sharply
A change to which fields the agent selected reduced exact match from 99% to 82%. The cause was that the agent recounted a truncated evidence list, so counts were computed on incomplete data. This is the aggregation problem appearing inside the agent itself.
A parsing bug and a stale artifact
A parsing bug mishandled one temporal question, and a stale benchmark artifact contained five incorrect counts. The write-up says regression tests were added for these failures. Those tests are the author’s and were not run here.
What this does not show
The benchmark is narrow. It covers one corpus, one set of 100 questions, one generation model, and one implementation. The write-up compares accuracy, tokens per question, latency, and generation-model calls. It does not provide a complete monetary cost model: there are no token prices, infrastructure costs, or break-even calculation, so “earn its cost” is answered in accuracy, speed, and call volume rather than in dollars. The exact-match gains also depend on gold answers that were prepared for this set. Results on your own data should be measured on your own questions.
Recommended Free Tools
Comparing approaches for your own task
If you run a similar comparison, measure at least these:
- Exact-match accuracy on representative questions, scored without a judge model, or with a judge checked against exact answers.
- Completeness of retrieved evidence, especially for counts and rankings.
- Latency per question, end to end.
- Tokens and generation-model calls per question.
- Monetary and infrastructure cost, which this benchmark did not model.
- How failures are detected and recovered, including silent failures that read as correct.
The source write-up is a DEV Community post by Anant Kumar from 2026. Its linked repository and dashboard were not independently inspected for this article, so the figures above should be read as the author’s reported results.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
