A low agent-evaluation score is evidence to investigate, not an automatic instruction to rewrite the prompt. First identify what the score measures, then inspect the interaction and its traces: the failure may be in the answer, tool choice, tool execution, agent guidance, semantic definitions, application behavior, instrumentation, or the test itself.
Krishna Tangudu’s account of debugging a Snowflake Cortex Agent offers a useful case study in separating those possibilities. The examples are observations and retests from one practitioner’s work—not a controlled benchmark or evidence of general performance. Read Tangudu’s account on DEV Community.
As an Amazon Associate I earn from qualifying purchases.
Start by asking what failed—and what the score measures
Two questions that sound similar can point to different evidence: “What was the score?” and “Did it answer the question correctly?” A score is only interpretable when you know its metric, expected behavior, and evidence source. A tool-selection score is not a percentage of correct answers; a correct final answer does not prove the agent used the required tool or followed a required interaction boundary.
Snowflake documents four system metrics for Cortex Agent evaluation. They examine different parts of an agent run, so use the metric that matches the behavior you need to assess. Snowflake’s Cortex Agent evaluations documentation also describes custom metrics judged by an LLM.
#1 Best Overall
- Package Includes: you will get 12 cute Christmas mini plush snowflakes, 4 styles, 3 pcs each, each snowflake is wearing a blue scarf and embroidered with a cute expression; Meet your Christmas gift giving needs or home decoration needs.Note: Because it is vacuum packed, you need to tap the plush snowflakes several times after receiving the goods, and wait for a few hours to return to its original state
- Lovely Design: these winter mini plush snowflake toys have snowflake shapes and cute expressions; They are all wearing scarves; Inspired by winter that captures the essence of winter; Create a warm atmosphere for your home as tabletop decorations
- Material and Size: mini plush snowflake are made of soft short plush fabric, filled with cotton inside, soft and smooth to the touch; Each small snowflake stuffed toy is about 4.3 inches in size, easy to carry and suitable for holding in your hand
- Ideal Christmas Gift: this Christmas mini plush toy is an ideal gift for any group of different ages, it can be given to friends, family, classmates, etc. They are widely suitable for winter theme parties, Christmas theme parties, birthday parties
- Wide Applied: mini stuffed snowflake toys are cute and novel, and can be applied for Christmas stockings or Christmas gift bag filling, Christmas decorations, weddings, birthday party gifts, classroom rewards, Christmas gifts, bringing a lot of fun to people
| Metric | What it examines | Ground truth required? |
|---|---|---|
| Tool selection accuracy | Whether orchestration invokes the expected tools. Extra calls can affect the score, depending on the expected and actual tool lists. | Uses expected tool behavior as a reference. |
| Tool execution accuracy | Tool inputs and outputs: whether the selected tool was executed appropriately. | Uses expected execution behavior as a reference. |
| Answer correctness | Whether the final response matches the ground-truth answer. | Yes; it evaluates against ground truth. |
| Logical consistency | Consistency across instructions, planning, and tool calls. | No ground-truth answer is required. |
| Custom LLM-judged metrics | Domain-specific criteria defined for a particular evaluation. | Depends on the custom metric and its evaluation setup. |
These measurements are complementary, not interchangeable. A low tool-selection result may flag an unexpected route even when the answer is sound. A high answer-correctness result, in turn, cannot establish that a required tool ran or that the agent paused for confirmation.
Trace the failure to its layer before changing anything
In Tangudu’s example, an agent failed to find an object that existed in metadata as a source consumed by other views. Adding a fallback instruction by itself did not fix the lookup. Inspection showed that the semantic tool’s definition included a source dimension, while its SQL-generation guidance emphasized searches by view name. The eventual change adjusted both the agent’s fallback instruction and the semantic-view guidance; a retest recovered the object and its downstream consumers.
That result supports a bounded conclusion: the revised lookup worked in that retest. Finding downstream consumers did not establish how every upstream object was loaded, and the observation is not a general claim about Cortex Agent performance.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
More broadly, a failure can arise in several places. Treat these as hypotheses to test against the case evidence, not reasons to make a change by default:
Rank #2
- What You Will Get: you will receive 18 pieces of winter plush toy snowflakes, there are 6 colors of these stuffed snowflake scarf, 3 pieces per color; Sufficient quantity and various styles, sufficient for your personal use needs and party needs
- Snowflake Design: each snowflake Christmas stocking stuffer for kids has been carefully designed and looks realistic and cute, it comes in different colors; Each plush soft snowflake for classroom has smile face on its face; If you own it, you will be happy
- Suitable Size: winter plush soft snowflakes for kids are about 3.9 inches/ 10 cm, These cuddly snowflakes are ready to snuggle down for the holidays; Ideal for indoor and outdoor use; You can use them in the way you need, creating an unforgettable moment for your loved one
- Reliable Material: the plush snowflakes are made of plush fabric, it feels very soft and comfortable; With nice workmanship and delicate stitches, they are soft, durable, comfortable to touch, not easy to break, deform, or wear out, suitable for long time use
- As a Christmas Gift: looking for cute Christmas stocking stuffers or Christmas carnival prizes? These cute plush snowflakes, with their blushing smiles, look lively and charming, they are very suitable as gifts; Use them as party decorations or party favors at your yuletide party, or hand them out to students at your child's winter
- Instructions: the agent may lack a fallback, clarification rule, or explicit stop condition.
- Tool routing: orchestration may select an unsuitable tool, omit a needed one, or make unnecessary calls.
- Tool execution: inputs, outputs, or tool behavior may not support the requested task.
- Semantic definitions: descriptions or SQL-generation guidance may direct retrieval toward the wrong names or dimensions.
- Application delivery: the end-to-end experience may fail a requirement even when an isolated tool check passes.
- Evaluation design: expected behavior, ground truth, or scoring configuration may not represent the intended task.
- Instrumentation: the application’s counters may not reflect the activity visible in other evidence.
Use batch evaluation and production traces for different jobs
Snowflake distinguishes evaluation from production observability. Batch evaluation scores an agent against a dataset before or after deployment. Monitoring helps debug and audit production conversations and traces. These approaches answer different questions: a dataset gives repeatable scored cases, while a trace shows what happened during a particular interaction.
Snowflake’s monitoring documentation describes production traces organized into turns and spans, with events for planning, tool calls and execution, SQL execution, response generation, and user feedback. Use traces to investigate the path behind a result; do not treat a score as a substitute for inspecting a case. See Snowflake’s guidance on monitoring Cortex Agent requests.
Tangudu also encountered an instrumentation discrepancy: an application tool-call counter treated missing metadata as zero, while native traces showed activity that the counter had missed. That is one observed discrepancy, not evidence that every application counter is faulty or that native logs are always complete. If a count conflicts with a trace, inspect what each source records before drawing a conclusion.
Free tools Windows power users keep installed
One-click scans. No signup required.
Test the behavior that must happen—not just the answer
A similar-name lookup exposed a different kind of failure. The agent retrieved a plausible candidate and began analysis without confirming the choice; the user had to correct it. Although a tool check later retrieved candidates successfully, an application retest still showed the agent proceeding without the required confirmation. The component could retrieve candidates, but the complete interaction still violated the requirement.
Rank #3
- This plush is approx. 5" x 3.5" x 4.5" in size
- Made from high-quality materials for a soft, fluffy touch.
- Fits in the palm of your hand!
- Own the whole #palmpalsparty collection!
- Holds bean pellets suitable for all ages to ensure quality and stability.
The desired behavior needs an explicit boundary: show the candidate objects, ask the user which one they mean, and stop before lineage or column analysis until they answer. A synthetic test pattern can encode that boundary without pretending it reproduces the production case:
- Provide a query with an ambiguous or similar object name.
- Require the agent to present plausible candidates and ask the user to choose.
- Require it to stop before lineage or column analysis while the choice is unresolved.
- After the user selects a candidate, check that subsequent analysis uses that selection.
Score the final answer and inspect the interaction path separately. An answer that happens to analyze the intended object does not demonstrate that the confirmation requirement was followed.
Write regression cases with independently checked expectations
Real user questions are good candidates for a regression set, but a question alone is not a complete test. Preserve conversational context when a follow-up depends on an earlier turn. Independently verify expected answers rather than treating a past successful response as truth: data can change, answers can become stale, and some questions should permit uncertainty instead of demanding a single unsupported result.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →For each case, specify both what should happen and what must not happen. Preserve working examples as well as the failing one so a targeted change does not silently break a workflow that already worked. When questions or expected answers change, treat the results as a new baseline; the difference cannot be attributed solely to the agent.
Rank #4
- Shimmering Design: In her shimmering snowflake cape, this Elsa plush doll captures the magic of Frozen. Perfect for your collection of Disney Princess toys, it's a must-have for Disney plushy fans!
- Dazzling Details: With metallic sparkles in her eyes, rosy cheeks & braided hair, this Elsa doll brings the enchantment of Disney princess dolls to life. Ideal for any collection of Elsa toys!
- Enchanting Outfit: Featuring a metallic bodice and satin skirt, this plush figure toy is the ultimate addition to your Disney toys collection. The perfect for Elsa toy for girls who love Frozen!
- Soft Plush Construction: This stuffed princess doll is made for cuddling, with its soft plush build and embroidered features. A perfect choice for plush toys lovers & fans of plushies for girls!
- Magical Adventure: This Elsa stuffed doll promises wintry dreams of adventure. This plush toy makes a great gift for girls who adore Disney dolls. Pair with the 14" Anna Plush doll, sold separately.
Also distinguish whether a capability ran from whether the answer was right. Tangudu enabled a Python sandbox but found no evidence in the inspected traces that it had been used, including for XML-related tests. A correct answer obtained through another route does not prove that Python was invoked. For a capability-specific case, inspect invocation and output evidence, then retest the same requirement through the real application and its actual tools.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Change the test only when its expectation is independently wrong
Tool-selection scoring can penalize extra calls, and Tangudu found that some expected tool lists omitted prerequisites required by the instructions themselves. That is a reason to inspect the case and scoring setup—not to relax a test merely because the agent failed.
Revise an expected route only when there is an independent reason: for example, a verified acceptable alternative, a documented prerequisite, or a corrected test case. Keep the desired user-visible behavior distinct from a particular implementation path when multiple routes genuinely satisfy the requirement. Conversely, if a tool call or confirmation step is mandatory, encode it explicitly rather than allowing a correct-looking final answer to conceal its absence.
Tangudu describes an illustrative, invented tool-selection example: if a test expects one tool call and the agent makes four calls, only one of which matches, the described formula gives 0.25. This is hypothetical arithmetic, not an observed score or benchmark result; it should not be read as a performance statistic.
Best Value
- What You Will Get: you will receive 24 pieces of winter plush toy snowflakes, there are 6 colors of these stuffed snowflake scarf, 4 pieces per color; Sufficient quantity and various styles, sufficient for your personal use needs and party needs
- Snowflake Design: each snowflake Christmas stocking stuffer for kids has been carefully designed and looks realistic and cute, it comes in different colors; Each plush soft snowflake for classroom has smile face on its face; If you own it, you will be happy
- Suitable Size: winter plush soft snowflakes for kids are about 3.9 inches/ 10 cm, These snowflakes are ready to snuggle down for the holidays; Ideal for indoor and outdoor use; You can use them in the way you need, creating an unforgettable moment for your loved one
- Reliable Material: the plush snowflakes are made of plush fabric, it feels very soft and comfortable; With nice workmanship and delicate stitches, they are soft, durable, comfortable to touch, not easy to break, deform, or wear out, suitable for long time use
- As a Christmas Gift: looking for cute Christmas stocking stuffers or Christmas carnival prizes? These cute plush snowflakes, with their blushing smiles, look lively and charming, they are very suitable as gifts; Use them as party decorations or party favors at your yuletide party, or hand them out to students at your child's winter
Keep evidence, hypotheses, and retest claims separate
A disciplined debugging cycle prevents one successful check from becoming a broader claim than the evidence supports:
- Retain the relevant evidence. Save the question and conversational context, expected behavior, score and metric, tool inputs and outputs, application behavior, and available trace.
- Investigate the specific failure. Determine what the metric assessed and compare it with the actual answer, route, tool execution, and user-facing interaction.
- Identify the layer to change. Separate observed facts from hypotheses about instructions, semantic definitions, tools, delivery, instrumentation, or test expectations.
- Write the regression behavior. State the expected action and any forbidden action, including required confirmation or tool use.
- Make the narrow revision. Change the component supported by the evidence rather than rewriting unrelated instructions or weakening the test reflexively.
- Retest at the right levels. Use a component check for the capability and an application retest for the complete interaction; inspect traces when the path matters.
- Record what the result establishes. Keep metric movement, case-level behavior, and answer quality as separate claims.
For comparisons over time, associate each run with the agent identity and version, skill revision, semantic-view definition, dataset, and scoring configuration. Without that context, a score change can be difficult to attribute.
What these cases establish—and what they do not
The examples show why agent evaluation is diagnostic work rather than prompt tuning by reflex. In one case, changing both fallback guidance and semantic-view guidance preceded a successful lookup retest. In another, candidate retrieval worked in a tool check but the application still skipped confirmation. A separate instrumentation discrepancy showed why a counter should not be accepted without understanding what it measures.
Recommended Free Tools
These are practitioner observations, not a controlled benchmark. They do not establish an aggregate improvement, general reliability rate, or broad comparison between types of users. The useful question before a fix is the one Tangudu proposes: “What should the agent do differently when someone asks this again—and what evidence would convince me it did?”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
