Who should write tests?

Before getting into who should write them, we need to talk about what kind of tests we're writing. The answer could be different when you're talking about unit tests vs end-to-end tests.

Unit tests make sure that individual classes and methods are working as intended, and it almost always makes the most sense for developers to write the unit tests while they code (if not before as in test-driven development). If you are a developer and you don't write unit tests, then you really are missing out. Unit tests improve the design of your existing code, actually speed up development --- not slow it down, and help you make sure you're not pushing broken code to your other team members. It's simply the polite and professional thing to do.

End-to-end tests (sometimes called automated acceptance tests) on the other hand, exercise the complete system from a customers point of view. While the Testing Pyramid recommends you should have fewer of these tests and they are usually slower to execute than unit tests, they can also be some of your most critical tests. These are tests like "Does our purchase path work?" or "Can new users sign-up for our platform?" If these mission critical flows don't work, you aren't doing your job and it's embarrassing and potentially damaging to your credibility and career. So who should be responsible for end-to-end tests?

Separate Dev and QA Teams

The traditional approach is to have separate Dev and QA teams. While this can work, it has some challenges that need to be managed.

First, you need to have a very strong relationship between leadership of the Dev and QA teams. If this relationship becomes adversarial, quality goes down and finger-pointing goes up. It is very toxic for an engineering organization if teams start getting into things like "My code is good, your test is bad." "We know how to write code, they don't", "They refuse to fix anything we find", etc. Having separate teams often creates an us-versus-them mentality and a diffusion of responsibility. not good.

Second, separate teams creates throughput and scheduling challenges. What happens when the Dev team starts completing stories faster than the QA team can test them? Either the QA team becomes the bottle neck and Dev time is wasted or the QA team cuts corners and misses bugs due to inadequate testing. What happens when the QA team starts completing stories faster than the Dev team (yes, this happens)? Either the Dev team becomes the bottleneck and QA time is wasted, or the Dev team starts cutting corners and not fixing bugs. In practice, this almost always results in unfixed bugs (either found internally or by customers).

Third, separate teams create "silos" of information. Devs know the code and architecture really well, but don't know how to test it. QA knows how to write tests, knows the functionality extremely well, and knows what areas don't work --- but is usually out of the loop on what changes are being made and what new tests should be written. Many of these challenges can be addressed with cross-functional teams.

Introducing Cross-Functional Teams

In this approach, Dev and QA team members are separate roles, but on the same cross-functional team (usually a "Feature Team"). What's nice about this approach is everyone is working from the same queue of priorities. It resolves some of the challenges of having separate teams around capacity planning, priority conflicts, scheduling conflicts, finger-pointing, and things like that.

This approach can also work well if your team has a "matrix structure", where people report in two directions: one Functional (i.e. Dev, QA, Infrastructure) and the other Feature (i.e. Product Owner / Program Manager / GM).

The challenges in this approach (particularly if not doing a matrix structure), is that the teams often go off and solve the same problems in different ways. While that's great for innovation, if left unchecked in can result in horrible inefficiencies. And when none of the teams know who really owns solving the problem, they can get into arguments about how to do it that are difficult to resolve.

In the testing world, this looks like Team A deciding to use Cypress and Team B deciding to use pyTest. Or Team A running tests on every PR while Team B only runs tests after deploying to a QA environment.

Simply Making Developers Own QA

In theory, this sounds great. No silos. No scheduling challenges. No finger-pointing. No diffusion of responsibility.

And this model can work quite well for quite a while. I was on a team doing Developer-only QA and we did test-driven-development and pair programming, shipped code multiple times a day, and relied very heavily on automated tests to be the only thing between the developers and production for a system with millions of daily active users. What's not to like about that?

Well, in practice, Developer-only QA often results in no one being responsible for owning the test orchestration problem. No one is responsible for really learning test automation and the tools in-depth. They are probably capable, but never have enough time. No one is building the systems required to run tests reliably and at scale. No one is working with vendors to bring in better systems. And every time any one wants to do any of these things, they are trying to get resources from teams that are constantly expected to focus on features or monitor production --- and managers rarely want to pull developers away from feature work or pull infrastructure folks off production to build out QA infrastructure.

Moving Towards Developer Enablement

This last approach seems to be becoming more common, especially as organizations grow and the complexity of their testing needs increases. In this model, you have more experienced SDETs whose primary role is not simply to write tests, but is instead to enable developers to own testing. They create the frameworks, work with vendors to bring in the right tools, manage the testing infrastructure, monitor the system under test. It's part developer, part devops, part tester, and part coach. They've made the leap from being test writers to being test orchestrators.

Specialization vs Generalization

Ultimately this is the classic specialization versus generalization problem. To what extent do you want developers to specialize (i.e. UI, API, QA, Infrastructure) versus generalize (i.e. full-stack).

Teams that have only specialists have people who are true experts in the technologies they work with. These are the teams that have people who have UI experts who keep an eye on the latest updates to Ember.js, Postgres experts who know what keys to use for sharding, and SDETs who know who to get xvfb running in a Docker container to record Selenium tests. But they're also often teams with organizational silos, diffusion of responsibility, scheduling/priority/capacity conflicts, and so on.

Teams that have only generalists have people who can handle any story that comes into their queue. They know how to do everything from adding a field in React, creating a controller in Spring Boot, creating database tables, and writing Selenium tests. But it's hard being good at everything and there are a lot of problems that deserve a specialist (e.g. How do we optimize retrieval from a table w/ 500M rows? How do we reduce our end-to-end test run time from 4 hours to just 20 or 30 minutes?).

Pragmatic managers will seek to have some balance between the two extremes.

So what's the right approach for your organization?

You can't simply pick one of these approaches without considering your team structure. While the developer enablement approach seems to work really well for organizations that are ready for it, getting there could require restructuring your teams, providing the them with the necessary training, and various other concerns depending on the size of your organization. While the results are often well worth it, we recommend seeking some expert advice before going down this route.