It's Time to Change How We Define Software Quality

A few years back my family was invited to stay with our extended family for a wedding. The night we arrived we were shown to our room, which was a beautifully prepared guest room that looked like it could have been the on the cover of a home design magazine.

As we were settling in, we saw a hand-written note next to the bed. It said something to the effect of, "Welcome to the Zoom Room! Turning on the set top box and TV can be a bit of a challenge. Follow the steps below. Kick off your shoes, lay back, and relax! We hope you enjoy your stay." While this may seem a bit over-the-top, the steps were surprisingly complicated (something like, "Push the blue button labeled 'A' to turn on the audio."). So it was actually very helpful to have instructions.

As we were unpacking, I realized I didn't have any shampoo. Even though I had been flying every other week for work for over a year now and felt I had mastered both the art and science of traveling, I had come to rely on hotel shampoo instead of bringing my own. When I went to the shower, what did I find? A complete set of hotel soaps / shampoos provided for guests. Another handwritten note let us know we were welcome to use them. I was really impressed!

So what does all this have to do with software quality? Well, that week I was about to kick off a quality improvement initiative and was trying to come up with a better definition of quality. Having realized I had just been through a quality experience, I started wondering. Why did I feel this was good quality? I thought about some other good and bad experiences I have had and did the same exercise. I was surprised what I found. Let's see if we can replicate this experience for you.

Take a moment to think about a time in your life when you've experienced good quality. Write down the reasons why it was good quality. Do the same for a poor quality experience. Write it down. Include the reasons why you thought it was poor quality.

Now take a look at what you've written for your good quality experience. Did you write more than just "the requirements were met" and "I didn't have to complain to anyone"?

Did you also write reasons like, "I didn't have to ask. My needs were anticipated before I even knew I would have them." or "I felt really important." or "I could tell someone really cared and spent time making sure I was comfortable." What do you notice about these reasons?

Now look at the reasons why you had a poor experience. Did you write more than "my requirements weren't met" or "something bad happened"?

Or did you also write reasons like, "When I told someone, they raised their voice and told me I was wrong." Or "My expectations weren't met and they refused to give me my money back"? Or "I know that's what I asked for, but I still didn't get what I needed. They didn't seem to care." What do you notice about these reasons?

When I did this exercise, I realized how much my observations about what was or was not good quality was tied to my own emotions and beliefs (e.g. my expectations about products and people, my past experiences, what I felt I needed). Even product attributes could be tied back to my own emotions and beliefs (e.g. How do you define "long lasting" or "high resolution" and why?).

It's time we take software quality beyond the checklist. It's entirely too convenient to define software quality simply as a checklist of requirements. To say that if the specified requirements are met by the code and all the tests are green, then we have quality. These checklists often don't even come directly from the customers but are often created by managers, product teams, testers, or even developers themselves. This Engineering-driven view of software quality treats quality as a property of the system-under-test. Something that is either true or false.

But real quality is so much more than simply a property of a system! Real quality is an emotion your customer feels when he or she interacts with your products and services. It's a feeling that expectations will not only be met but also be anticipated. It's a feeling of importance. It's a feeling of comfort and security. It's a common understanding of what will happen even when other things change. It's a belief that what's under the hood is good, even when we can't see it.

This may seem like an impractical definition. After all, if every customer gets to decide for themselves what quality means to them, how on Earth can we know if we've achieved it? But the practical consequences of thinking of quality in these terms is enormous and should not be overlooked.

Ultimately, we need to start measuring software quality by what people say and what people pay. So take a moment to think about what your customers and employees are saying and paying.

What are they saying? Do customers recommend your products and services to others? Do they speak highly of them on social media? Do they respond well on customer satisfaction surveys? What about your employees? Do your sales, marketing, and support teams speak highly of your products and services when there aren't any customers around?

And what are they paying? Do customers pay a premium for your products and services? Do you have high retention rates? And what do your employees pay? How much time do your employees spend providing work-a-rounds, answering questions about confusing offerings, and fixing the same mistakes over and over again? Do your server logs show evidence of issues happening but not being reported?

Since quality is more than just a simple property of the system, in turns out we may not be able to precisely measure it. And for some customers, meeting their definition of quality might not be obtainable or profitable even if we could. But for me, this expanded definition of quality is what makes ensuring quality exciting.

Quality is so much more than just having green tests. Through this expanded definition of quality, we have a real opportunity to make peoples lives better in a meaningful way and to rid the world of poor customer experiences. And who doesn't want that? Please join me in taking on this challenge!