This is a guest post by Peter G Walen.
Test automation has become ubiquitous, a bit like wifi. It is everywhere in software development. Sometimes, it is done powerfully and well, other times, it is suboptimal.
The basic premise is that automated testing, tests software faster, more often, and speeds up accurate delivery. Although this can be the case, It is not a guarantee.
Managers want automation, so testers jump to building frameworks, tests, and processes without analyzing the “why”. Developers want faster feedback on their code changes, so we get suites of unit and service tests wrapped up in continuous integration.
While we’ve written about test automation before in posts like Three Steps to Set Up an Automation Strategy and have spoken on related topics in past automation-focused webinars, it is time to ask the questions and challenge the common assumptions of “Why do we need automated tests?”
Common responses to why we need test automation
- Faster and more accurate development: Good test automation can speed up development thanks to the fast test execution and the repeatable nature of tests. Automation also helps eliminate human error, therefore reducing the risk of failure.
- Tests are included in the build process: Organizations can include tests in the build process and can be initiated each time a build is completed.
- Automated functional tests: Automated functional tests can be created by the development team while the application software is being developed and used to exercise the change in their local environment.
Challenging common responses
Each of these ideas is good. Together they present solid ideas for test automation. The focus is on definite goals that can show measurable, repeatable behavior and activity trends.
The challenge comes when we look beyond high-level goals and intent. There is a consistent effort in some circles and by some practice vendors, to associate Agile software development with automated testing and only automated testing. Such challenges are often created by people with a shallow understanding of testing, test automation, or testing tools. Some are blatantly presented as “facts” and “proven outcomes” by some unscrupulous tool vendor representatives which leads to a distortion of the goals and desires stated above.
Challenge 1: Automation makes everything faster
This is an interesting challenge. It often comes from a belief that Agile and/or SCRUM will always make delivery faster and quality better. What makes this interesting is that while those things may be true sometimes, it is not true in absolutely all instances.
At times, test automation can make things faster, however, there is no certainty that the total time to test will be reduced, nor that it will decrease time to delivery. There is no certainty that automation by itself will do anything faster, other than initiate consecutive executions.
While automation will help you initiate consecutive executions, it is only able to accomplish this in sequence, meaning that generally, only one automated test can be run at a time. By practicing parallel testing, automation allows different tests to run at the same time accelerating execution across multiple versions.
Under the right circumstances, building test automation code while the application code is being built can help testing to start as early as possible. Building both the test and application code together, in parallel, can help the people developing work together to make sure there are fewer unexpected surprises in calls, interfaces, log entries, etc.
Challenge 2: Test Automation means you can test better by replacing people
This challenge is brought to us by the people keeping accounts and budgets. Because the organization is spending “too much money” the obvious “cost savings” occurs by reducing headcount. That is, people. The people in question are usually the people who do “manual testing.”
After all, automated testing will make it so we won’t need anyone doing manual testing, and if we don’t get rid of “overhead” the organization will have a cost increase, right?
Yes. We won’t need people pounding away at manual scripts, as long as it makes sense to automate those scripts. However, will the people who remain understand enough about the product to help design good tests that go beyond simple unit tests? Will they be able to design full functional tests? What about maintaining the integrity of the regression suite and not just the coded scripts but the logic behind them?
It is common to see that the questions about understanding the system under test and fully understanding the automation code are often mutually exclusive. Thus, to maintain some rigor in testing, an organization might not be able to reduce the number of people by the expected amounts.
The skill of test automation has garnered much interest in the testing industry, so much so that it is now considered an in-demand skill without which a tester may become irrelevant. Many testers are scared of the fact that not being able to write code or test scripts would render them useless to their teams. But this is actually not the case.
What people tend to get wrong is that they focus exclusively on the idea of automation “replacing” human testers. Simply asking which humans will be replaced fails to account for how testing and test automation will evolve.
Whether manual or automated, to make any form of testing meaningful and valuable to the organization, thoughtful consideration is needed. The significance of the information produced can, and should, help drive the decisions around quality.
Creating a balance of human testing plus automated checks is the best way to get ROI on test automation efforts. Here’s how we can make an effort to strike that balance:
- Understanding what needs to be automated instead of running behind a coverage percentage or numbers.
- Automating only what is really needed and using human testing for exploration.
- Making conscious decisions rather than struggling with technical debt.
Challenge 3: One tool will fit all of your needs
This challenge, perhaps more than the others, causes much consternation among software development teams.
The “best tools available” can do some amazing things. Still, I’ve not seen a hammer or screwdriver trim a piece of 4×4 as needed. A hammer might sink a screw, not very well of course, but it could do it. Likewise, a screwdriver could, maybe, drive a nail. I suspect it would take a bit of doing and massive amounts of frustration, but one could, conceivably, get it done.
But why would you want to?
Similarly, why would you want to use a test automation tool for functions it was never intended or designed to be used for? Some tools are good for performance tests. Some are excellent for API-level tests. Others are built for UI tests.
Some of these tools might have secondary uses to broaden their reach. The challenge comes in learning them well enough to teach them to others and mastering them well enough to do what they are promised to do.
Why do we need test automation?
This brings us, finally, to the question. Why do we need test automation?
Let us consider tools, in general. Why do they exist? Do we need them? What do they do for us?
We can use our hands to dig in the soil, say to plant seeds or transplant a seedling. It might be more efficient to use a small trowel. If we need to dig a larger hole, a spade might be a better choice. A shovel could be used to dig an even bigger hole and shift small and medium-sized rocks as needed.
For some jobs, shifting soil with the fingers is just fine. For bigger jobs, tools make sense. Why use those tools? That is simple. They allow people to do the job quickly and get on to other tasks.
So the next time someone asks “Why do we need automated tests?” don’t just respond with high-level goals and intent, let them know that automated testing allows us to be freed of the mundane, hole-digging tasks of testing, so we can get on to the other tasks requiring deep exploration, creativity, intuition, and domain knowledge. This is the human aspect of testing.
Now that you know the answer to the “why” of test automation, you should know the “how”? That’s why we have compiled a list of questions to help you prioritize what you should automate next to guide your test automation strategy!
