FlakyTest.dev

What FlakyTest.dev is

FlakyTest.dev is a web service for Python teams that need to manage, monitor and report on test suite performance. Its pytest plugin tracks tests in Python projects, and test ownership directs flaky test notifications to the person who wrote the test. For an individual test, teams can review its runs; failing tests include debug details such as standard output and standard error. Run statistics can help identify tests that slow down builds.

Users can mute a test through the interface without making code changes or opening a pull request. A muted test continues to run and record errors, while no longer failing the CI build. The product includes flake detection and a quarantine workflow. Its setup guide documents a GitHub Actions configuration using a repository secret named FLAKYTEST_SECRET_TOKEN; the plugin sends test results when that environment variable is set. FlakyTest.dev currently focuses on Python projects, with other language support described as a future plan. Anže, the maker, describes himself as a software engineer and indie maker with over ten years of experience shipping products for Silicon Valley startups and growing development teams. The OSS plan is free for personal projects, non profits and open source projects. Business includes a 60-day free trial, with pricing afterward to be determined.

Who it suits

FlakyTest.dev suits Python teams dealing with unreliable test suites, especially teams that want to assign flaky test notifications and inspect individual test runs. Personal projects, non profits and open source projects can use the OSS plan; teams considering Business can start with its 60-day free trial.

Limits

The OSS plan is for personal projects, non profits and open source projects. Business pricing after its 60-day free trial is to be determined, and current language focus is Python.

Getting started

1. Visit flakytest.dev.
2. Choose the OSS plan if the project is personal, a non profit or open source, or start the Business 60-day free trial.
3. Use the pytest plugin in the Python project.
4. Set the FLAKYTEST_SECRET_TOKEN environment variable so the plugin sends test results.
5. For GitHub Actions, follow the documented configuration using a repository secret named FLAKYTEST_SECRET_TOKEN.

Verdict

Python teams managing flaky tests should consider FlakyTest.dev for ownership notifications, per-test history and muting that keeps errors logged without failing CI. Personal projects, non profits and open source projects have a free OSS plan. Teams needing another language or a settled Business price should look elsewhere or wait for those details.