Running your tests in parallel when building software is a necessity if you want high quality and fast feedback loops. It is required to attain continuous delivery in a timely manner.  Parallel test execution is a great way to reduce overall execution time making teams more efficient.  Just keep in mind that for best results there should be a strategy in how to write tests that will perform well when running in parallel. Consider best practices like:

  • Keep your tests small.  Keeping tests small will help your suite run efficiently and provide you with faster results.
  • Avoid dependencies between test runs.
  • Ensure each test uses its own test data and doesn’t depend on data created from a previous test.
  • Avoid using shared state across tests.
  • Consider tagging scripts to more easily run groups of tests.

What is parallel testing?

Parallel testing is running multiple tests at the same time in order to reduce the overall test execution time while testing an application across multiple machines. Parallel testing is harder to set up than sequential testing but so worth it.  Here is the difference:

Sequential Test Execution

sequential test execution

Parallel Test Execution

parallel test execution

Parallel testing is also different from concurrent testing. Concurrent testing is similar, but there is an important difference.  Concurrent testing is running multiple tests at the same time in order to reduce test execution time on the same machine.  This difference is important to understand because you do not have unlimited concurrent tests because you are limited by the machine you are running on.  With parallel testing you can spin up as many machines as needed to support your goals.

parallel vs concurrent test execution

Let’s show an example of a development team and the effect parallel testing can have.  Team Tiger uses a gitflow release strategy and has 2 developers working different features. Wait time happens innately every time tests have to be run.  For this example, assume there are 100 tests that each take 1 minute to run = 100 minutes when running sequentially. Both developers are pushing their feature to develop separately and the 2 features get put into a release together.

If tests are run sequentially:

  • Total wait time is 4 x 100 minutes = 400 minutes (3.6 hours)
  • This would only give Team Tiger 1 deploy, maybe 2 deploys per day if the first deploy was early morning.
  • Dev #2 waits 100 minutes for tests to complete from Dev #1 pushing to develop before he can push to develop.  Even though this developer could in theory move on to another task, experience has shown that most likely he will be context switching between any new task and getting this feature into develop, thus reducing his efficiency.

If tests are run in parallel with 10 concurrent test runs:

  • Total wait time is 4 x 10 minutes (100 / 10 parallel tests) = 40 minutes
  • This would give Team Tiger the ability to do more than 10 deploys per day.

Result: running with 10 parallel tests made Team Tiger 10X more efficient giving them more chances to deploy and get feedback quicker.

Why parallelize tests?

The obvious reason to parallelize your tests is time.  For instance, parallel testing and improving time will improve lead time, deployment frequency, change failure percentage, and even mean time to recovery if a team includes testing in that process. It also allows teams that have integrated tests into their CI/CD pipeline to execute in a timely manner to support continuous deployments.

Time is a big reason but not the only one.  Parallel testing also improves the overall quality of an application by allowing teams to run more tests faster throughout the CI/CD workflow. When tests are slow, teams are tempted to only run subsets of tests or no tests at all in order to get a deployment out. Parallelizing tests allows them to run all tests needed to ensure the highest quality.

Another reason to parallelize tests is to remain agile when change is needed.  Faster builds and deployments mean teams can make changes more often.  In the event a crisis happens and there is a production issue (or some other urgent event), teams can make the necessary changes and get a tested fix out more quickly than they could otherwise.

How to parallelize tests

When we talk about scaling tests by running tests in parallel, we are talking about the test environment (versus the product environment where the product under test lives). A test environment has 2 different ways you can scale: distributing tests and distributing browsers (in the case where you are running UI tests).

Scaling by distributing tests

In order to parallelize tests, something has to instruct the test framework how to break up the tests and run multiple test runners simultaneously.  Each test runner runs a subset of the overall group of tests. 

Scaling by distributing browsers

In order to parallelize browsers, something has to spin up browser instances that know how to execute browser instructions simultaneously. (This is only needed when performing UI tests)

Distributing tests is usually managed by one of the following ways:

  1. Test framework
  2. Continuous integration tool
  3. Tools that can automatically parallelize and scale tests

Distributing browsers is usually manage by one of the following:

  1. Tools that can launch multiple browsers
  2. Cross browser testing tools

Each of these approaches addresses the tests, test runners, splitting of the tests, and browser/api differently.  There are pros and cons to each approach.

Distribution of Tests

Let the test framework parallelize tests.  First of all, not all test frameworks support parallelization.  Only some frameworks have parallelization built in as an option.

When a testing framework manages the parallelization, a common setup is like the following diagram:

continuous integration

In this approach you tell the test framework how many parallel processes you want.  The test runner will spawn a new thread for the number you specify. Test frameworks that support parallelization like this make it fairly easy to add parallel testing into your process.  However, beware of common gotchas:

  1. The machine is limited by processing capacity.  After 2-3 threads per CPU you may start seeing degradation of test performance.  You may have to upgrade your machines significantly to support this parallelization approach.
  2. Only certain frameworks support parallelization.  See chart below for some examples.
  3. Make the tests you write thread-safe.

Here is a non-exhaustive list of some test frameworks and their parallelization support:

Test Framework

Parallel Testing Support

Reference

Cucumber

Cucumber-JVM allows parallel execution across multiple threads since version 4.0.0.

https://cucumber.io/docs/guides/parallel-execution/

Cypress

Cypress can now run tests in parallel across multiple virtual machines, or can be parallelized on a single machine. Cypress will automatically figure out, detect, and load balance all of your specs in the fastest and most efficient way possible. However, attempting to run in parallel on a single machine requires server grade hardware with dozens (or even hundreds) of CPU’s and GB of RAM to run efficiently.

https://docs.cypress.io/guides/guides/parallelization.html#Overview

JUnit

Opt-in feature since version 5.3.  Considered an experimental feature at the time of this writing (fall 2020).  Sounds like parallel test output is not great.

https://junit.org/junit5/docs/5.3.0-M1/user-guide/index.html#writing-tests-parallel-execution

Mocha

mocha-parallel-tests is a test runner for tests written withmocha testing framework which allows you to run them in parallel.mocha-parallel-tests executes each of your test files in a separate process while maintaining the output structure of mocha.

https://github.com/mocha-parallel/mocha-parallel-tests

NUnit

Supports parallel test execution since NUnit framework 3.0.

https://docs.nunit.org/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.html

Robot

Does not support parallel test execution, but there are modules you could add like Pabot to parallelize for you.

https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html

Specflow

Has parallel test execution, but not great support.

https://docs.specflow.org/projects/specflow/en/latest/Execution/Parallel-Execution.html

Test Project

On a single machine you can have a single agent. The way to run multiple jobs in TestProject is to create multiple agents (on multiple machines, they can of course be remote agents) and assign your jobs to those agents. It’s possible but not easy to use.


Webdriverio

Yes, spawns processes on the same machine when using maxinstances parameter.  Limited by CPU on the machine.

https://webdriver.io/docs/organizingsuites.html

If a framework doesn’t have parallel test execution support there might be a 3rd party module you can install that supports parallelization. Some of these modules have a fee.  Beware though, in many cases these modules create additional strain on your machine resource usage, so a misconfiguration can result in maxing out resources that ultimately slows your builds down or causes failures. A few examples of modules are:

Node:https://www.npmjs.com/package/mocha-parallel-tests

Rails:https://github.com/grosser/parallel_tests, https://github.com/ArturT/knapsack

PHPTest:https://github.com/brianium/paratest

Let the continuous integration tool parallelize your tests.  Build servers were not designed to be test environments which is the main reason why it has taken a LONG time for CI tools to even get to where they are today when it comes to parallelizing tests.  

Similar to how each framework works a little differently, continuous integration tools have a few differences in their support of parallelization as well. What is common is the CI pipeline launches tests across multiple agents that you will need to configure. You generally tell the CI how to split the tests by passing in test names or test suite names.

continuous integration pipeline

Sounds great right?  It is in the sense you can scale out parallelized testing using your build server.  However, there are some downsides to this approach that you should be aware of.

  1. There are unlimited machines available with this approach, but they can get costly and hard to build and maintain.  Using build server parallelization increases the number of builds which is expensive and it increases wait time for teams waiting for backed up builds.  Read the blog post “Don’t Pay Your CI To Run Tests” to see more details on this.
  2. Another downside to this approach is lack of consolidated test results.  There will be test results from each machine that you will want to consolidate to get an overall picture of quality.  This can be a pain to do yourself.

Here are a few continuous integration tools and how they handle parallelizing tests:

Appveyor

Test Results: has a Test tab to view results but only supports a few framework results by default (like xunit).  If you use any other framework you have to send results to API to view results in Appveyor

Limitations: maximum build job execution time of 60 minutes (it will cut off your test runs after 60 minutes!)

Pricing: 4 concurrent jobs = $200/month (4 concurrent tests)

https://www.appveyor.com/docs/parallel-testing/

Azure Pipelines

Test Splitting: Azure pipelines supports parallelization by running multiple agents.  There are 3 different ways of splitting tests.

Test Results: tests results resulting from runs across multiple agents can be cumbersome to view 

Pricing: There is a charge per job and agent.

https://docs.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops

CircleCI

Test Splitting: you can also can split tests by execution time

Uniqueness: use docker containers to parallelize tests for a more scalable approach. Defaults to 80 concurrent jobs without paying for more

Pricing: pricing is a little hard to understand and manage, using L Linux server = $15 for 1250 testing minutes

https://circleci.com/docs/2.0/parallelism-faster-jobs/

Codeship

You can parallelize as much as you want to - the only barrier is the resources on the host machine.

Pricing: 4 concurrent builds, 3-5 parallel tests per build = $600/month

https://documentation.codeship.com/pro/continuous-integration/parallel-pro/

Gitlab

Parallelization uses docker containers

Capped at 50 parallel jobs


TravisCI

Pricing: 5 concurrent jobs = $228/mo (annually)

These jobs wouldn’t account for build and testing time though


One of the biggest concerns when relying on a continuous integration server to handle parallelization is cost.  Continuous integration tools weren’t initially built with parallelization in mind.  Most tools charge per parallel job which can get costly.

Tools that can automatically parallelize and scale your tests.  We only have one tool in this category so far, and that tool is Testery.  The reason why this tool is in a category on its own is because it can do the following that other tools can’t do:

  1. Automatically scale the test runners
  2. Automatically split the tests
  3. Manages the test infrastructure for you
  4. Reduces load on your build server
  5. Consolidates the test results into single view and sends back test result to CI (or anywhere else you want it)
  6. Doesn’t have the performance issues (lag) that tools using Selenium Grid have
parallel test execution with Testery


Distribution of Browser [instructions]

Another way of scaling web based tests is by having one master test execution machine launch multiple Web UI tests that each use a remote Selenium WebDriver. Instead of distributing tests (which is still needed for full parallelization) this approach distributes the browser instructions.  This makes it pretty easy to test the same functionality using multiple web browsers.

Cross browser testing tools.
Cross browser testing is a great strategy to make sure that the web sites and web apps you create work across an acceptable number of web browsers.  Selenium Grid is a solution that was created to allow for parallelized cross browser testing.  Selenium Grid is a smart proxy server that makes it easy to run tests in parallel on multiple machines. This is done by routing commands to remote web browser instances, where one server acts as the hub.  Selenium Grid doesn’t actually run the tests so you still need something to run the tests.

Here is how Selenium Grid works:

selenium grid

Using Selenium grid allows you to run parallel tests using different browsers and operating systems. Selenium grid allows you to mimic real user input, support all major languages, and supports many browsers and frameworks among other benefits.

The drawbacks of using a Selenium grid is how the technology is based on http protocol.  Tests run on a client machine and sends each browser instruction across the wire causing slowness. Http wasn’t intended for long requests and very often tests have to wait for something to happen.  A load balancer may kill requests because its waiting causes issues producing flaky tests.

With Selenium grid you only have access to those platforms, browsers and browser versions on which your nodes are running and eventually the performance of Selenium grid degrades drastically with the larger number of nodes you connect to via hub.

Remember, with all these tools you still need to manage your test runner and ensure parallelization is turned on correctly.  Make sure to pay attention to the load these tools are putting on your CI/CD servers. Cross browser testing tools can cause heavier load than intended causing your costs to spike or processes to fail.

Most cross browser tools in the market today are based on and use Selenium Grid. Here are few of the options:

AWS Device Farm

Does NOT use selenium grid.  (the only one in this list that doesn’t)

Mobile focused - they say web but only support mobile testing frameworks

Pricing: $250/concurrent device per month, unlimited usage

https://docs.aws.amazon.com/devicefarm/latest/testgrid/what-is-testgrid.html


https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types.html

Bitbar

Test Automation offers the possibility to test mobile apps and websites instantly and effectively using test automation frameworks.

Pricing: charges by parallelization (up to 15 parallel)

https://bitbar.com/mobile-app-testing/

Browserstack

Pricing: charge by parallel test (2 parallels = $259/month, 5 parallels = $650/mo)

https://www.browserstack.com/question/581

CrossBrowserTesting

Pricing: Live manual testing (5 parallel tests = $100/mo), Automated testing (5 parallel automated tests, unlimited minutes = $500/mo)

https://crossbrowsertesting.com/selenium-testing/grid,
https://crossbrowsertesting.com/pricing

LambdaTest

Web and mobile test automation platform utilizing a slew of testing frameworks

Pricing: pay per parallel test (5 parallel = $500 / month)


Limitation: run into issues with queues if you set your parallelization to more than what you are paying for and could lose tests

https://www.lambdatest.com/support/docs/junit-with-selenium-running-junit-automation-scripts-on-lambdatest-selenium-grid/

Perfecto

Parallel test execution in perfecto is running tests across different platforms (i.e. Chrome vs Safari) or relying on open source testing frameworks parallel execution functionality.

Pricing: Automated testing ($125+/month)

https://www.perfecto.io/blog/parallel-testing

SauceLabs

Limitation: As tests complete, queued tests are allocated to concurrency slots in the order they were queued, however, we do not recommend queuing tests intentionally. Tests will time out with an error if they are queued for too long and/or if too many tests are already queued.

Pricing: automated testing (5 parallel tests, unlimited test minutes = $749/mo)

https://saucelabs.com/pricing

As you can see, there is a lot to consider when wanting to do parallel test execution.  If you would need any help or just have questions feel free to reach out to Testery.