An incident-response agent can score perfectly while answering questions about incidents it has already memorized. That tests retrieval, not whether it can help with an unseen incident. In a redo, I held out 10 cases, compared the agent with and without its memory, and got a much more informative result: 9 of 10 correct with memory, versus none fully correct in the baseline. The difference is striking, but this small, single-run evaluation does not prove the agent will perform reliably in production.
The first 10/10 measured lookup, not unseen-incident performance
My initial test used 10 incidents that were also stored in a Hindsight memory bank. For each, the agent found the associated root cause, warned about the trap action, and cited the incident. The result was a perfect score—but the test cases were already in the agent’s memory.
As an Amazon Associate I earn from qualifying purchases.
That overlap changes what the score means. A system that retrieves a relevant stored postmortem may be useful, but success on that task does not show that it can diagnose a novel incident. As I put it: “If the test data is in memory, you’re testing lookup.”
Free tools Windows power users keep installed
One-click scans. No signup required.
How the held-out test was set up
For the redo, I used 114 OpenSRE incidents. I retained 104 in a fresh Hindsight memory bank and held out 10. I wrote queries for the held-out cases using symptoms rather than copying their root-cause language, then ran each query under two conditions:
#1 Best Overall
- Memory condition: the agent had access to the memory containing the 104 retained incidents.
- Baseline: the same model and prompt were used, but the memory block was removed.
I graded responses against the dataset’s true_category field. I counted a response as a partial match when it named a plausible cause in the right area but got the mechanism or trigger wrong. Outputs were saved in eval_holdout_results.json.
What the two conditions returned
| Condition | Fully correct | Other reported outcomes |
|---|---|---|
| Memory available | 9/10 | One query blocked by Groq’s daily rate limit; I counted it as a miss. |
| Memory removed | 0/10 | Four partial matches and six responses I classified as hallucinated. |
These are my results from this particular 10-case evaluation, not an independently replicated benchmark. The comparison suggests that memory helped on these queries, but it does not establish how the system would perform across other incidents, models, or operating conditions.
Why the baseline comparison matters
A memory-enabled score alone cannot show what the memory contributed. The baseline offers a reference point by keeping the model and prompt the same while removing the memory content. That makes the difference easier to interpret as an outcome of these two setups, though it does not isolate every factor in the full system or prove memory was the sole cause.
One demo query described checkout-service 500 errors after a deployment. In the no-memory response, the model invented a NullPointerException, reported log counts from a kubectl command it had not run, and referred to a Helm revision that did not exist. The memory-backed answer instead suggested a dependency-capacity issue and cautioned against rollback based on similar incidents. It also proposed irrelevant network and systemd checks, so having memory did not make every part of the response useful.
The demo’s confidence label was extracted from response text with a regular expression. It was not a calibrated probability and should not be treated as one.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What this evaluation doesn’t show
The large gap is worth investigating, but several design limits constrain what can be concluded:
Rank #4
- Only 10 held-out incidents: a small sample can give an unstable picture of performance.
- One run per query: the test says nothing about run-to-run variation.
- One grader: I graded the responses, so there was no independent check of the classifications.
- Category-level grading: matching
true_categorydoes not establish that the agent identified the exact event or mechanism. The partial-credit rule also involves judgment. - Related cases may remain: held-out incidents came from the same dataset and vendor set as the retained incidents; the test did not use a deliberately different population.
- One rate-limited run: the memory-backed set included a query blocked by Groq’s daily limit, which I counted as a miss. That is transparent, but it mixes service availability with diagnostic accuracy.
- No component ablation: the test did not separate the effects of reflection, recall, trap boosting, and signature enrichment.
Accordingly, the scores do not establish repeatability, performance on substantially different incidents, or which individual memory-related component produced the observed difference.
Recommended Free Tools
Quick Recap
Best Value
A more defensible way to test an incident agent
- Choose the holdout before building memory. Keep evaluation incidents out of the memory bank from the start so the test measures behavior on cases the agent cannot retrieve directly.
- Write symptom-only queries. Describe what an operator would observe without copying the postmortem’s diagnosis, category, or distinctive root-cause phrasing.
- Keep the baseline controlled. Use the same model and prompt instructions in both conditions; remove the memory content for the baseline.
- Set grading rules in advance. Decide what counts as an exact category match, a partial match, or a wrong answer. Separate plausible general area from correct mechanism and trigger.
- Record failures as well as answers. State how timeouts, rate limits, and other unavailable runs are handled. Preserve the raw outputs so readers can inspect what the score represents.
- Repeat and broaden the test. Use more cases, run queries repeatedly, involve an independent grader, and select a holdout more distinct from the retained cases before making broad performance claims.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
