Test Automation: What Is It Good For?
Many technical managers and executives believe a large suite of automated tests is critical to engineering a “high-quality” software product. While I’d always rather have automated tests than not have them, I nonetheless consider that viewpoint to largely be a misconception as it relates to “quality”, and today I’m going to take a minute to explain why.
Notice that I called out “technical managers” and “executives” as being excited about test automation, and not product owners. That’s because
Product owners don’t care if you have automated software tests.
They really don’t, not directly anyway. Product owners operate on a different level. They are concerned about two things in this context:
Lack of failures in production.
Ability to react to changing business requirements.
Out of the gate, automated software tests do nothing for concern #1. To see why, let’s assume a base level of self-discipline on the part of our software developers and think this through.
If a developer builds a new feature, they are going to try it out before they throw it into production. (Yes, assumption, but if it doesn’t hold then the software engineering manager or lead has some up-front work to do prior to thinking about test automation.) If the developer finds a bug upon trying out the code, they will fix it, at which point it ceases to be a bug, and again since this is happening before the code goes to production, this bug can never manifest as a failure for the user. That makes for a happy product owner.
We don’t write automated tests to find bugs, at least not initially.
Whether the developer automated the steps that they took to find that bug matters nothing whatsoever to the product quality the day that code goes to production. At this point, the only thing test automation does is provide documentary evidence of some subset of things that the developer tried out. That’s not nothing, but it’s not a huge boon to product quality, either.
So why do we care about writing automated software tests? To answer that question, we need to move to concern #2. What happens when we need to change the software in response to end-user requests or changing business requirements?
Anytime you touch code, you run the risk of breaking something. If our self-disciplined developer implements a new feature in an existing product, they will try out that new feature (see above). And if they’re somewhat familiar with the product and the code, they may also try out some scenarios with related features. But most likely, over time they will start to lose the ability to anticipate all the possible areas of breakage.
Our engineering organizations used to employ professional test engineers who were generally better than most developers at anticipating where bugs may manifest during modifications. Back in those days, we often managed to maintain a “high quality” product over a long period of time with little or no test automation — hence my original statement in the first paragraph of this post.
But most companies got rid of test engineers over the past 10 or 15 years in the name of cost savings Agile and DevOps. The developers that remain can take a lot of extra time to validate the software before sending it to production, but they are under time pressure and (Agile and DevOps notwithstanding) often still don’t see testing as primarily “their job”.
So we’re left at higher risk of what we call regressions slipping through to production and impacting our users. You probably see where I’m going with this:
The purposes of test automation are to speed up validation and prevent regressions.
That’s it, full stop. We write test automation so that we can minimize the time it takes to do a full re-test of an existing system when sending updated software to production, and maximize the chances of finding bugs in existing functionality while doing it.
In fact, I wish we more often used the terminology “automated regression testing” to make this relationship clear.
Test automation is certainly an important part of the modern software development process, but we should make sure we understand why we’re doing it.
Feel free to reach out and let me know how you’re using test automation in your organization, and if you’d like to hear more about this subject!
