Review AI-generated SQL as a draft: define exactly what the number should measure, check how the query gets it, then validate the result against trusted data before using it. A query can run successfully and still answer the wrong question. Microsoft warns that generated queries may be inaccurate or return results the user did not intend, and Databricks makes the distinction plainly: “Execution success is not the same as correctness.”
1. Define what the number is supposed to mean
Before reading the SQL, turn the request into a testable specification. Write down the population, measure, exclusions, time range, and output grain—the level represented by each result row. For example, “monthly sales by customer segment” should specify which sales count, which customers qualify, how cancellations are treated, which dates define the period, and whether the output should have one row per month and segment.
As an Amazon Associate I earn from qualifying purchases.
Give the model approved table and column names, documented relationships, and the organization’s definition of the metric. If “active customer” or “revenue” has a local meaning, provide it rather than asking the model to infer it. Ask it to list its assumptions and source tables along with the SQL. Databricks recommends treating generated SQL as a reviewed draft, not evidence by itself: guidance on trustworthy AI-generated SQL.
2. Trace the query from source tables to output
Check every referenced object against the approved schema, then follow how rows are filtered, joined, grouped, and calculated. A query may be syntactically valid while using the wrong table grain or relationship and producing a plausible but inflated total.
#1 Best Overall
Tables, columns, and joins
- Confirm that each table and column exists and is appropriate for the requested metric.
- Verify each join key against a documented relationship; do not assume similarly named columns mean the same thing.
- Check whether either side has duplicate keys at the join level. A one-to-many or many-to-many join can multiply rows and overstate sums or counts.
- Compare row counts and distinct business keys before and after important joins. Unexpected growth is a signal to investigate, not proof of an error by itself.
Filters, groups, and dates
- Confirm that the grouping matches the requested grain. Extra grouping columns can split one expected result into several rows; missing ones can combine distinct populations.
- Check whether each filter is applied at the right stage. A condition in a join can behave differently from one in a
WHEREclause, particularly with outer joins. - Verify the date column, time zone where relevant, inclusive or exclusive boundaries, and calendar definition. A fiscal period is not necessarily a calendar month.
- Inspect how nulls, zero values, and missing periods are handled. For ratios, check both the numerator and denominator and ensure division by zero has an intentional treatment.
These checks matter because a query can return rows without using the intended population, grain, join, or calendar. Databricks’ guidance on text-to-SQL for business decisions also distinguishes a successful execution from a correct answer.
3. Validate the result independently
Do not use the generated query’s own output as its only validation. Compare it with a trusted dashboard, approved query, or reviewed source for the same metric, population, dates, and exclusions. A reference is useful only when its scope matches; differing filters or calendars can explain a mismatch without proving which result is wrong.
- Preview representative records. Inspect a small, bounded set of underlying rows and trace a few results through the calculation.
- Reconcile totals. Compare the generated result with an independently trusted figure for precisely the same scope.
- Check structural signals. Review total row count, distinct keys at the expected grain, null rates in important fields, and unexpected duplicates.
- Test edge cases. Examine empty periods, boundary dates, missing values, zero denominators, and categories with few records.
- Investigate discrepancies. Identify whether they arise from scope, join multiplication, definition differences, or calculation logic before accepting either number.
Databricks’ SQL validation guidance describes checks such as comparing results with trusted data and inspecting representative records. GOV.UK’s general guidance adds an important caution: “Generative AI systems are fallible and the correctness of their responses is not guaranteed.” See AI Insights: Generative AI.
4. Limit what the query can do
Use access limited to the task and approved data. Review write-capable statements and follow the organization’s approval process before execution; the ability to generate SQL is not a reason to grant broader database access.
Rank #3
Permissions and defaults differ by tool. Microsoft’s SSMS Copilot documentation says queries run under the connected user’s security context and do not elevate permissions. It describes a read-only default, while noting that DML or DDL can run when the user’s permissions and configured mode allow it. Those details are specific to SSMS Copilot, not a general guarantee for other AI SQL tools; check Microsoft’s SSMS Copilot documentation for current product details.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.5. Put a person in charge of consequential results
Require an accountable owner to approve both the SQL and the output when a wrong number could affect financial reporting, customer communications, or automated decisions. Approval should cover the metric definition, query logic, and reconciled result—not merely whether the query ran.
Rank #4
Keep the request, generated SQL, and result together so the decision can be reviewed later. Maintain a set of representative questions with reviewed expected results, and rerun them when relevant schemas, metric definitions, or models change. Databricks discusses these controls in its text-to-SQL workflow guidance; Microsoft’s product-specific transparency note likewise warns that generated responses may not be accurate or intended: Transparency Note for Copilot in SSMS.
Quick Recap
Best Value
A compact review checklist
- Is the population, measure, exclusion set, time window, and expected grain explicit?
- Are the tables, columns, join keys, and metric definitions approved?
- Do joins preserve the intended row counts and distinct keys?
- Are grouping, filter placement, calendar boundaries, nulls, and zero denominators handled correctly?
- Do representative rows and totals reconcile with a trusted reference of matching scope?
- Is execution restricted to approved data and appropriate permissions?
- Has an accountable person approved consequential uses, and can the result be retested later?
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
