Analyzing flaky tests with Testery

Sometimes no matter how hard we try to isolate and keep tests stable, there can be times were flaky tests creep in during infrastructure changes or be inherited from legacy projects. Testery has great features which help identify your current flaky tests and track them over time until they are under control.

What are flaky tests?

Flaky tests in the context of the Testery platform refers to tests which fail when running against an environment, but pass later on a re-run in the same context.

Prerequisites

In order for flaky tracking features to be used automatically, the project must have the "Retry Failed Tests" enabled under project settings.

💡Note: Your tests and environments are in excellent shape if they don't need to retry to pass. Use caution when enabling this option when there are no signs of flaky tests, as it can lead to an increase of test time if re-running real product failures.

💡Note: If you manually re-run a failed test and it passes, this will also detect as flaky.

Test Runs

For a quick look at recent flaky tests, test runs will display the number of flaky test detected after the run is complete.

Analytics

For a more in-depth look at flaky tests across your projects, head over to the Flaky Tests section of the Analytics Page.

Here you can filter on multiple fields and get some stats on your flaky tests. This is especially useful to identify which tests are flaky and tracking trends over time.

More on the topic of flaky tests:

Leading causes of Flaky tests

What is a Flaky Test? A flaky test is an automated test that validates inconsistent behavior, generating distinct results when implemented multiple times on similar functionality. Such tests are known for their unpredictability, as they can fail or pass intermittently without any alterations to the code or the application under

Testery BlogNick Shatskyi

Quarantining Flaky Tests on the Testery Platform

Flaky tests can be a challenge for your team to manage. At Testery, we give you the tools to manage test flakiness using the flaky test quarantine features. Check out the demo video below to see how it works. Start a Free Trial On Testery To keep our runs reliable

Testery BlogAdam England