A CTO Guide to Scaling SpecFlow for Enterprises

SpecFlow is a fantastic way for organizations to encourage collaboration in testing. The team-based approach to quality it encourages produces better quality software. But you know this. You're here because you've got so many SpecFlow tests that now you need to get them to run faster.

Whether you want to make sure your team isn't waiting around for test runs instead of cranking out features, or you just want more visibility into how the tests are performing, this guide will walk you through the steps required to get your tests humming along nicely.

Determine a strategy for parallelizing your test runs. There are various approaches here. You can have tests for multiple projects running at the same time. You can split the tests by suite or by feature file. The fastest and most scalable option here is to do scenario-based parallelization.

Determine where the tests will run. The easiest way to run tests is on localhost, and the next logical step is getting them off localhost and running in your CI/CD system. And this works well for a while. But while most CI/CD systems support running various types of tests, they ability to do horizontal scaling at the scenario level.

As the number of tests increase, you inevitably find your team doing "creative" things like setting up separate builds for separate test suites. It's a lot of work doing this balancing act. The tools simply weren't built to parallelize at the individual test level. A better option is to leverage a purposely built continuous testing platform.

Determine how test results will be stored. If your team is building your own parallelization mechanism in-house, they will need to determine how test results will be stored. One option is to report the test results back to your CI/CD system. This works well when the tests are running single or multi-threaded on a single CI/CD agent. Another better option is to store the test results in S3, a non-relational database, or leverage a platform with this capability.

Figure out how you will review history and perform analytics. An important part of keeping maintenance costs low for your test suite is to be able to review history and analytics across test runs. Which tests are always failing? Are the test runs speeding up or slowing down? If you choose to build your own, you'll need to select a reporting platform compatible with your test results storage.

Don't forget monitoring and alerting. Monitoring and alerting aren't just for production! Your test suites should be considered a production system whose job it is to determine whether or not your pre-production systems are ready to go live. Any systems and infrastructure should be monitored and alerts should go out before issues happen (e.g. before your agents run our of disk space). Failing to do this can result in the whole team being blocked while the testing infrastructure is fixed.  

Don't forget the non-SpecFlow tests. Getting SpecFlow tests running well is great and all, but why stop there? More and more teams are relying on multiple forms of testing (unit, integration, performance, security, etc.) Having all your tests running optimally and having your test results available in one place is key to getting the most value out of your testing.

If you're looking for a continuous testing platform that was purposely built for running SpecFlow tests at scale with these considerations in mind, we hope you check out Testery.