Test a SQL agent on two separate questions: did it retrieve the right result, and did its written answer stay within what the database supports? SQL text similarity alone cannot establish correctness, and one matching result on one database can hide a flawed query. A reliable evaluation combines execution-based checks, realistic application cases, explicit tests for uncertainty, and a review of the benchmark’s reference answers.
What should a SQL-agent test measure?
Evaluate the whole path from question to response, not just whether the generated SQL runs. A useful test plan distinguishes query validity, result correctness, and the truthfulness of the explanation shown to a user.
As an Amazon Associate I earn from qualifying purchases.
- SQL validity: Does the query parse and run in the intended database dialect?
- Result correctness: Does it return the right rows and values for the question?
- Answer grounding: Does the natural-language response accurately describe the returned rows, without adding unsupported facts?
- Appropriate uncertainty: Does the agent clarify, qualify, or abstain when the available schema or data cannot answer the question?
Record syntax failures, execution errors, wrong results, timeouts, and abstentions as distinct outcomes. Combining them into a single pass/fail score makes it harder to see what needs fixing.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchHow do you build a representative test set?
Start with the application’s actual questions and schema
Public benchmarks help with comparable baselines, but they cannot substitute for cases drawn from the tables, terminology, and workflows your users actually encounter. Include questions that cover:
#1 Best Overall
- Simple lookups, filters, sorting, and aggregates.
- Joins, duplicate rows, null values, and empty results.
- Date ranges and boundary conditions, such as whether an end date is inclusive.
- Values the agent must copy from the question or retrieve from data.
- Multiple steps or queries, if the product claims to handle workflows.
- Different phrasings of the same request and different amounts of schema context.
Include straightforward questions alongside realistic edge cases. For ambiguous wording, missing time ranges, or conflicting metric definitions, define the intended behavior in advance: the agent may need to ask a clarifying question rather than guess.
Use benchmark scope that matches the product
Spider 2.0 is a reference for enterprise-oriented text-to-SQL work: its tasks involve real-data environments, large schemas, multiple dialects—including BigQuery and Snowflake—and operations spanning transformation and analytics. The original Spider 2.0 paper introduced 632 real-world text-to-SQL workflow problems in 2024. The benchmark site’s currently listed settings differ by task type: Spider 2.0-Snow has 547 examples, Spider 2.0-Lite has 547, and Spider 2.0-DBT has 68 code-agent tasks. The displayed setup lists Snow and DBT as no-cost and notes that Lite can incur cost; verify current requirements before planning an evaluation.
Benchmark scores are tied to a particular task setting and evaluation snapshot. The Spider 2.0 paper reported 17.0% of its tasks solved by an o1-preview-based code-agent framework, compared with 91.2% on Spider 1.0 and 73.0% on BIRD in that paper’s setup. Those figures describe that experiment, not a general or current score for those models.
How do you check whether the SQL is actually correct?
Use execution-based evaluation, but do not stop at one database
Exact SQL string matching is a poor sole definition of correctness: logically equivalent queries can use different syntax or structure. Conversely, an incorrect query can return the expected result by coincidence on a particular database instance.
Execution accuracy checks whether a predicted query produces the expected result on a database. Test-suite accuracy strengthens this check by comparing query results across a compact set of databases designed to distinguish likely incorrect queries. Zhong and colleagues’ 2020 study reported that their distilled Spider test suite distinguished more than 99% of generated neighbor queries in that study. That result describes their test-suite construction experiment; it is not a guarantee for every benchmark, database, or agent.
The published test-suite evaluator supports execution or test-suite accuracy as well as exact set match. Its documentation describes test-suite accuracy as the reported metric for the official Spider, SParC, and CoSQL leaderboards, with exact set match retained as a reference. Configure the evaluator for the task: for example, its documentation includes a value-plugging option for systems that do not predict values.
Keep exact match as a diagnostic, not the verdict
Report the selected execution or denotation metric and, when useful, exact-match results alongside it. A gap between the two can help identify cases where SQL form differs while results agree, or where a candidate’s apparent similarity does not lead to the right result. State the evaluator and its configuration so readers can tell what “accuracy” means.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How do you catch unsupported answers and test abstention?
There is no canonical unsupported-answer metric established by the sources cited here. Define and score this behavior in the application’s own evaluation set rather than treating it as part of SQL execution accuracy.
Best Value
Include questions the available evidence cannot answer
- Ask for a field or measure that is absent from the schema.
- Ask for a causal conclusion when the database contains only descriptive observations.
- Use ambiguous terms, conflicting metric definitions, or omit a necessary time range.
- Include empty-result cases and cases where the rows provide only partial evidence.
Score answer behavior separately
For each case, specify whether the acceptable response is to answer, ask for clarification, state the evidence limitation, or abstain. Track unsupported factual assertions, missed opportunities to abstain, unnecessary abstentions, and correct answers as separate outcomes. For answerable cases, compare the explanation with the actual returned rows; a correct query does not guarantee a faithful summary.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can the benchmark’s reference answer be wrong?
Yes. Reference SQL, question wording, and expected output formatting all need review; a gold query is not automatically ground truth. In a 2026 analysis, Jin and colleagues identified annotation issues in 80 of the 121 Spider 2.0-Snow examples for which gold queries had been released. Their examined issues included date-boundary mistakes, joins or flattening that inflated row counts, incorrect join keys, and ambiguous output formatting. The 80-of-121 finding applies to that reviewed subset, not to all Spider 2.0 tasks or SQL benchmarks.
When an agent’s result disagrees with a reference, inspect both before assigning blame. Check the question’s intended meaning, schema, expected output, and reference query’s joins, filters, types, date boundaries, and duplicate handling. Where possible, have a reviewer adjudicate the case and retain the correction and rationale so the same defect does not keep distorting results.
Free tools Windows power users keep installed
One-click scans. No signup required.
What should an evaluation report disclose?
Accuracy numbers are meaningful only in the context of the task, data, and evaluator used. Use the following checklist when publishing or comparing results:
| Setting | What to record |
|---|---|
| Task scope | Single query, conversational turn, multi-query workflow, or code-agent task. |
| Database and schema | Domain, schema size, table and column hints, and whether oracle tables were supplied. |
| Dialect and environment | Target engine and SQL dialect, plus relevant environment constraints. |
| Metric | Exact set match, single-database execution accuracy, test-suite accuracy, or another named measure; include evaluator settings such as value handling. |
| Reference quality | Gold-query provenance, how ambiguity is handled, and any known corrections. |
| Operating behavior | How abstentions, clarification requests, unsupported explanations, errors, and timeouts are recorded. |
| Reproducibility | Benchmark revision, data version or snapshot, random seed, agent configuration, and repeated-run policy. |
Spider 2.0 notes that results may change as metric accuracy is checked and examples are updated. It also calls for identifying methods that use ground-truth tables as an oracle-table setting. Do not compare leaderboard values as if they were a controlled experiment until the task setting, hints, metric, and evaluation date match.
Quick Recap
A practical evaluation sequence
- Define the product claim. Decide whether the agent is expected to answer one-off SQL questions, support conversation, or complete multi-query workflows.
- Assemble cases. Combine a relevant public benchmark with application-specific questions, edge cases, ambiguous requests, and questions the available data cannot answer.
- Set expected behavior. Review intended results and specify when the right response is an answer, clarification, limitation statement, or abstention.
- Run the correct evaluator. Match the SQL dialect and task setting; measure execution or test-suite accuracy and use exact match only as a companion diagnostic where useful.
- Inspect disagreements and failures. Separate SQL and execution problems from reference-data defects and unsupported natural-language claims.
- Publish the conditions with the score. Disclose benchmark revision, data snapshot, schema hints, oracle-table use, metric configuration, and environment constraints.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
