Analyzing flaky tests with Testery
Sometimes no matter how hard we try to isolate and keep tests stable, there can be times were flaky tests creep in during infrastructure changes or be inherited from legacy projects. Testery has great features which help identify your current flaky tests and track them over time until they are under control.
What are flaky tests?
Flaky tests in the context of the Testery platform refers to tests which fail when running against an environment, but pass later on a re-run in the same context.
Prerequisites
In order for flaky tracking features to be used automatically, the project must have the "Retry Failed Tests" enabled under project settings.
💡Note: Your tests and environments are in excellent shape if they don't need to retry to pass. Use caution when enabling this option when there are no signs of flaky tests, as it can lead to an increase of test time if re-running real product failures.
💡Note: If you manually re-run a failed test and it passes, this will also detect as flaky.
Test Runs
For a quick look at recent flaky tests, test runs will display the number of flaky test detected after the run is complete.
Analytics
For a more in-depth look at flaky tests across your projects, head over to the Flaky Tests section of the Analytics Page.
Here you can filter on multiple fields and get some stats on your flaky tests. This is especially useful to identify which tests are flaky and tracking trends over time.