Analyzing flaky tests with Testery

William Gunn

May 13, 2024 • 3 min read

Sometimes no matter how hard we try to isolate and keep tests stable, there can be times were flaky tests creep in during infrastructure changes or be inherited from legacy projects. Testery has great features which help identify your current flaky tests and track them over time until they are under control.

What are flaky tests?

Flaky tests in the context of the Testery platform refers to tests which fail when running against an environment, but pass later on a re-run in the same context.

Prerequisites

In order for flaky tracking features to be used automatically, the project must have the "Retry Failed Tests" enabled under project settings.

💡Note: Your tests and environments are in excellent shape if they don't need to retry to pass. Use caution when enabling this option when there are no signs of flaky tests, as it can lead to an increase of test time if re-running real product failures.

💡Note: If you manually re-run a failed test and it passes, this will also detect as flaky.

Test Runs

For a quick look at recent flaky tests, test runs will display the number of flaky test detected after the run is complete.

Analytics

For a more in-depth look at flaky tests across your projects, head over to the Flaky Tests section of the Analytics Page.

Here you can filter on multiple fields and get some stats on your flaky tests. This is especially useful to identify which tests are flaky and tracking trends over time.