A failed test is evidence to investigate, not proof that the product is broken. A useful anomaly report preserves the run context, shows whether the failure is new or recurring, points toward a cause, assigns an owner and records how the fix was verified.
What a test anomaly report should do
A test anomaly report turns an unexpected result—such as a failure, timeout, blank page or intermittent pass—into a reproducible investigation. It should help a teammate answer three questions: what happened, under what conditions, and what should happen next.
As an Amazon Associate I earn from qualifying purchases.
Do not assume every failed test indicates a product defect. Microsoft identifies several possible sources, including the code under test, faulty test code, environment problems and flaky behavior. Azure DevOps Test Analytics supports reviewing failure details and execution instances; its features are specific to Azure DevOps.
What to include in a test failure report
Capture enough detail that someone who did not observe the run can find the same result and compare it with other runs.
- Test identity and outcome: test name or ID, suite, expected result and actual result.
- Execution context: build or release, branch, commit or change set when available, run time, and environment or agent details.
- Reproduction evidence: steps, relevant inputs and test data, and whether the failure reproduces on rerun.
- Failure details: error message, stack trace, logs and relevant attachments such as screenshots, where available.
- Analysis and ownership: suspected category, next action, assignee, status and links to related bugs or work items.
Azure DevOps test-run documentation describes run summaries, linked work items, step outcomes, automated-run stack traces, analysis information and attachments. See Run automated tests. A screenshot can make a browser failure easier to understand, but it should supplement—not replace—logs, steps and run context.
How to investigate a failed test
- Preserve the original result. Save its details and attachments before editing the test, rerunning it, or changing the environment. Link it to a bug or work item if the workflow supports that.
- Check its history. Compare multiple executions across a useful time window. Look for the first failure, prior passes, recurrence, and whether failures cluster around a build, branch, environment or time period. Azure DevOps Analytics offers top-failing-test views and drill-down into execution instances; available views and defaults are product-specific.
- Classify the likely source. Start with four buckets: product code, test code or data, execution environment or infrastructure, and nondeterministic behavior. The categories can overlap; an unreliable dependency, for example, can make a test fail without a defect in the code the test was intended to assess.
- Test the hypothesis. Inspect relevant changes and rerun in a controlled way. When order or shared state may matter, run the test independently. Record what changed between runs rather than relying on a single pass as proof of resolution.
- Choose a remedy and assign it. Fix the underlying cause where possible, or create and assign a product defect when evidence points to one. Record the owner and next step so the report does not end at diagnosis.
- Verify the remedy. Run the affected test after the change and watch later executions for recurrence. Keep the history visible so a repeated failure is not mistaken for a new, unrelated incident.
Why tests pass sometimes and fail other times
Flakiness can arise in test logic and data, setup and cleanup, the runner, the application and its dependencies, or the operating environment. Google’s testing guidance recommends checking assumptions and test data, setup and teardown, and whether tests behave differently when run independently. See Flaky Tests at Google and How We Mitigate Them.
State, order and cleanup
A test may depend on data left by another test, an unreset database, a reused browser session or cleanup that did not complete. Initialize required state explicitly and clean it up reliably. Run the test alone and in a different order to see whether the failure follows shared state or ordering.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchTiming and synchronization
A test that checks too early can fail before the application reaches the expected state. Synchronize on a meaningful condition, such as a selector or completed action, rather than inserting an arbitrary delay. Google cautions that fixed sleeps can both slow tests and remain flaky when system timing varies.
Resources, dependencies and environment
Check runner capacity, network and service availability, dependency responses, and environment configuration. Capture access times or relevant logs where possible. A failure limited to one agent, environment or period may point away from a product-code change and toward infrastructure or an external dependency.
Choose a fix without hiding the signal
- For test logic or data problems: make inputs explicit, isolate state, and remove reliance on test order.
- For setup or cleanup problems: initialize prerequisites and ensure teardown runs even after a failure.
- For timing problems: wait for application state or a specific condition instead of guessing with a fixed delay.
- For runner or environment problems: address the resource limit or uncontrolled environmental assumption and retain evidence of the conditions.
- For a confirmed product defect: create or link a defect, assign an owner and record severity and status under the team’s process.
- For repeated reports with one root cause: connect duplicates to the underlying issue instead of counting every report as a separate defect. ISTQB defect-management material describes retaining one report when multiple reports share a root cause; confirm the applicable syllabus edition for formal process requirements. ISTQB Certified Tester Foundation Level.
Marking a test as flaky can help distinguish known instability from a new regression, but it should not erase the evidence or become a substitute for investigation. Microsoft documents workflows for detecting, marking and later unmarking flaky tests; the designation affects future executions rather than retroactively changing the current pipeline result. See Flaky test management.
Use screenshots as supporting evidence for browser failures
For a UI test, a screenshot can show what the browser displayed at the failure point. Keep it with the test result alongside the URL, run identifier, timestamp, logs and test steps. A screenshot is a visual record of one moment; it does not by itself explain whether the cause was application code, test timing, a consent banner or an environment issue.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server. A single GET request captures a URL; its clean-shot handling accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups and chat widgets before capture. These steps can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and the response identifies the page verdict and billing status in headers.
Use this cURL example to save a screenshot of a page for review:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo documentation for API parameters. The response can be PNG, JPEG, WebP or PDF. Its MCP server offers take_screenshot, get_page_info and capture_pdf for Claude, Cursor and other MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots.
Sign up for 1,000 free screenshots a month—no card required.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #4
Compare reporting workflows by what they preserve
Tool choice matters less than whether the workflow supports investigation and follow-through. Evaluate these capabilities against your team’s needs:
- Evidence depth: Are steps, stack traces, logs, screenshots and attachments available?
- History: Can an investigator compare multiple executions and identify first failures or intermittent patterns?
- Traceability: Can the result connect to a requirement, bug, branch or code change?
- Flake handling: Can known intermittent tests be tracked without losing history or obscuring new regressions?
- Ownership: Can analysis, status, severity and next actions be assigned and revisited?
Azure DevOps documentation illustrates one product’s analytics, run-evidence and flaky-test workflows, but these sources do not establish a vendor-neutral ranking of test-reporting tools. Choose based on the evidence and traceability your team actually needs.
Cost and reliability considerations
When a report depends on a screenshot, capture reliability affects whether the artifact is useful, but the image should remain one piece of evidence in the test record. Distinguish a failure to load or capture from a genuine application assertion failure; record the verdict and run details so an infrastructure issue is not filed as a product defect. For ScreenshotNeo, response headers identify the page verdict and whether a capture was billed, and cache hits are not billed.
Best Value
Google’s 2016-era article reported that about 1.5% of Google’s test runs were flaky and defined a flaky result as one that both passes and fails with the same code. That is a historical, Google-specific observation—not a current or industry-wide flakiness rate. The practical lesson is to use a test’s own execution history rather than infer the cause or prevalence from one run.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Frequently Asked Questions
Should I rerun a failed test before reporting it?
Preserve the original result and its details first. A rerun can help test a hypothesis, but it should not replace the initial evidence.
How can I tell whether an intermittent failure is a product defect?
Use execution history and controlled reruns to narrow the source; intermittent behavior alone does not establish whether the cause is the product, test, runner, dependency or environment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
