Flakiness.io provides test analytics for GitHub and GitLab, tracking results across commits to help teams distinguish flaky tests from regressions. It classifies a test that switches between passing and failing on the same commit as a flake; a persistent outcome change at a commit is treated as a regression. Listed runners include Playwright, Pytest, Vitest, Jest, CucumberJS, Bun, Rust and a JUnit XML bridge, with a Node.js SDK for custom integrations. The service says it works with any CI provider, merges shards, and separates staging and production histories. Reports include test history, grouped errors, system telemetry and a test waterfall; artifacts can include logs, screenshots, videos, traces and HTML files. Playwright can balance shards using recorded test durations. The Free plan lists 1 GB of artifact storage and 90 days of data retention. Startup costs 49.00 USD per month and includes 10 GB and 180 days of retention. Self-hosting is also available, using PostgreSQL and S3-compatible storage.
Who it is for
Flakiness.io suits software teams tracking test behavior across commits in GitHub or GitLab. It also offers a self-hosted option for teams that want to deploy the application themselves.
What is good
- Identifies flakes and persistent test regressions.
- Supports a range of test runners.
- Separates staging and production histories.
- Reports include errors, telemetry and test waterfalls.
- Free plan includes unlimited test runs and users.
What to know first
- Free plan retention is 90 days.
- Free plan includes 1 GB artifact storage.
- Self-hosted licensing details require contacting the maker.
Verdict
Flakiness.io combines cross-commit test analysis with reports and uploaded run artifacts. Consider its storage and retention limits, or ask the maker about self-hosted licensing.
Flakiness.io plans and pricing
All plansCompared on test analytics software
- Free plan
- Yes
- Paid from
- $49/mo
- Supported frameworks
- Playwright, Pytest, Vitest, Jest, CucumberJS, Bun, Rust, JUnit XML
- CI/CD integrations
- GitHub Actions, GitLab CI/CD
- Flaky-test detection
- Yes
- Failure analysis
- Yes
- Execution trend reports
- Yes
- Data retention
- 180 days
- Self-hosted deployment
- Yes
