Tests First, Tools Second

How Did We Get Here?

I love software testing. It was my first really significant job in the industry, and it's been a passion of mine ever since. Throughout my long career, I’ve always been an advocate for testing, emphasizing its importance at every opportunity. There’s something uniquely satisfying about ensuring that software is reliable, robust, and ready for the real world. Testing is where theory meets practice, and it’s a critical part of delivering quality software.

So we need to have a conversation…

In today's fast-paced software development world, the drive to ship features quickly has often overshadowed the fundamental principles of effective software testing.

This shift is especially evident in companies that have eliminated full-time professional software test engineers, offloading the testing responsibilities onto developers. While this approach might seem efficient — or at least cost-efficient — on the surface, it has led to a troubling trend: The rush to automate testing without first understanding what actually needs to be tested.

The Lure of Automation

Developers by nature are problem solvers. When faced with a challenge, their first instinct is to write code, build tools, and automate processes. This Amazon-style “bias for action” is a strength, but when it comes to testing, it can also be a weakness. The attraction of shiny new testing frameworks, automation scripts, and CI/CD pipelines often draws teams into the trap of creating an extensive suite of automated tests without fully considering the purpose of those tests.

This is a recipe for disaster.

Without proper planning, these automated tests can end up focusing on the wrong areas — particularly the areas that happen to be easiest to automate — leaving critical parts of the application untested. The result? A false sense of security, where everything appears to be working—until it doesn’t.

Caveat

Just to be clear, I’m talking about functional testing today, not unit testing. This might surprise some of you, but unit testing is primarily a coding, not testing, activity. I’ll write more about that in a future post.

The Forgotten Art of Test Design

When I started as a test engineer at Microsoft in the mid-90s, I was taught a different approach to testing. (Yes, Microsoft actually had an in-house professional training organization doing this kind of thing back then.) Even though we were probably the most advanced automated-testing organization in the world, we didn’t jump straight to automation. Instead, we began with the basics: Collaborating with developers and program managers,, understanding the business requirements, and using them to design test cases.

These test cases were carefully recorded in a dedicated test case management tool, where thousands of tests were documented, tracked, and organized to ensure comprehensive coverage of the application.

(Real test case management tools are far more sophisticated than simply using Excel and allow you to track, organize, and manage the testing process in a structured manner. Sadly, today, it’s rare to see such a tool put into practice, with many teams relying on simpler methods or skipping formal test case management altogether.)

This process of deliberate, thoughtful test case design is something we’ve lost sight of. It’s easy to see why: Manual test case creation and review can feel slow and cumbersome compared to the immediate gratification of automation. But without this crucial step, we risk automating for automation’s sake (or for the sake of test coverage metrics on a report somewhere), rather than ensuring our tests are actually meaningful.

Risk-Based Testing: A Strategic Approach

One of the key benefits of taking the time to design test cases is that it allows us to engage in risk-based testing. Not all parts of an application are created equal. Some features are more critical than others, and some bugs would have a far greater impact on users if they were to slip through. By prioritizing our test cases based on the risk associated with different areas of the application, we can ensure that our testing efforts are focused where they matter most.

Risk-based testing also allows for more intelligent use of resources. Instead of trying to achieve 100% test coverage—an often unrealistic and almost always misguided goal—we focus on covering the most critical code paths and use cases. This approach leads to more efficient testing and better outcomes for the engineering team as well as the customers.

Once you’ve designed your test cases using a risk-based approach, this is where automation starts to enter the picture. With a clear, well-structured set of test cases, automation becomes an incredibly valuable tool for executing repetitive tests efficiently and consistently.

And yes, while this might seem like heresy to some, it's actually possible to release a well-tested, high-quality software application without any automated testing at all—provided the testing problem is approached correctly. This approach involves meticulous test case design, prioritizing testing efforts based on risk, and ensuring comprehensive coverage through manual testing.

Shift-Left Testing: The Earlier, The Better

To further improve testing efficiency and quality, consider adopting a shift-left testing approach. This involves integrating testing earlier in the development process, preventing defects from propagating downstream and reducing the overall cost of testing.

The Path Forward: Tests First, Tools Second

So, what’s the solution? It’s simple: tests first, tools second. Before writing a single line of automation code, we should take a step back and ask ourselves what we’re trying to achieve with our tests. What are the business requirements? What are the risks? What are the most critical scenarios and user journeys that need to be tested?

Only after we’ve answered these questions and designed our test cases should we start thinking about tools and automation. This approach might take a bit more time upfront, but it will pay dividends in the long run. We’ll end up with a test suite that not only provides real value but also helps ensure the quality and reliability of our software.

Remember: Test automation is never the end goal; it’s simply a method to achieve the ultimate objective of delivering a high-quality product.

Conclusion

In conclusion, while tools and automation have their place, they should always be in service of well-designed, risk-based test cases. By focusing on what really needs to be tested, we can ensure that our testing efforts are meaningful and effective.

By focusing on what really needs to be tested and adopting a strategic approach, we can ensure that our testing efforts are meaningful and effective. If you're looking to enhance your testing strategy, I'd love to discuss how to refine your process, develop a robust test case design and management strategy, prioritize the right tests, and ensure your software meets the highest quality standards.