This is a guest post by Justin Rohrman.
Most of the software projects I have worked on come along with either a specification or a set of acceptance criteria. To the product owner, this is a list of what to build. To me, it also looks like a list of test ideas.
A tester can take each point in the specification, or each line item in the user story, perform that as a test, and be done. That creates a nice, tidy picture of testing that is easy to describe to non-technical people, as well as easy to get done in a timebox.
As a strategy, it can be a good start, but it’s a bit like a swimming pool with two inches of water: too shallow.
Abandoning the idea that the specification is the start and end of the testing work is an improvement, but it also brings more challenges. Without a list of criteria, how do we know when testing is done — or how to get there?
Complete testing is impossible
Imagine a very simple webpage to help someone approximate how much shipping will cost. That single-page app has the following controls:
- A set of radio buttons with one for < 10 miles and the other for > 10 miles
- A checkbox if the delivery person has to walk up flights of stairs
- A checkbox for deliveries that occur on the weekend
- A Submit button to perform the calculation
- A label to display the calculated rate
How many tests do you need to perform to completely test this page? One? A hundred? A thousand? You could build a truth table and see that there are eight or nine possible configurations of radio button and checkbox combinations (if you count no selections). But is that really complete testing?
I’m sure that you have guessed the answer is no, so let’s dig into some other possible important test ideas.
There are a few questions I would ask, starting with the design. What happens when delivery is exactly 10 miles and is miles the right unit of measure here? What counts as a flight of stairs? I see that there is an upcharge for deliveries made on the weekend, but what about holidays or evenings, or rush deliveries?
Those are design questions, and a lot of them could be answered by learning more about where in the development process this product is, who the person using it is, and what they expect. Next, I would attack some ideas that involve interacting with the product.
This is a big one: what platforms are supported. One platform is easy, but these days, one platform is rarely the answer. On the desktop, we have Windows, macOS, possibly some flavors of Linux, and likely several versions of each. Mobile devices have Android, iOS, and a number of minor players. Apple’s iOS generally has one main version in use at a given time, while Google’s Android may have hundreds or even thousands because of the resources available to the developer community. The product may behave just a little differently in each of these environments.
But that’s just the tip of the iceberg. What about data storage? What about emailing quotes? What if a user spams the Submit button — does that create false orders or make the page crash? This isn’t an article on test ideas, but we could come up with five, or 10 even, and just be scratching the surface.
Knowing that there is an infinite number of tests that could be performed for any one field, let alone for a whole page, we have to decide when to stop.
Heuristics for stopping testing
Every time I test a new code change, I also make a decision about when I should stop and move on to the next task. Usually, that decision gets made using a rule of thumb, or heuristic, like the ones below.
I’m out of time
Software development always happens on a time scale. Moving from idea to production takes a couple of hours or a couple of weeks. Arguably, the most popular way to develop software is within a two-week sprint. By the time it is Friday on the second week, there is some set of features the team agrees are done, maybe some that have not yet been started, and some that are still being worked on. At that point, we might decide to either move the in-progress work to a new branch and release it in the future, or to release it with the knowledge that there might be some problems. But, for now, testing on those changes is done.
In the context of a sprint, the idea of “out of time” gets a little fuzzy. If testing is becoming a bottleneck because it’s taking too long, we might be out of time. If testing this one story is taking so long the entire sprint might be late, we might be out of time. If I am testing the first story of the sprint, I may need to estimate how long all the rest of my testing might take, in order to determine whether we are pushing the limits of time availability. We rarely use a stopwatch to decide if a story is taking too long; instead, these things are decided with a conversation, often during a standup meeting.
I’m not finding bugs anymore
Testing is sometimes like wringing out a soaked cloth. You fold it a couple of times and give it a twist, and a lot of waterfalls out. Fold it once more, twist again, and less water comes out. Fold it one last time, twist, and you might get a drop if you are lucky.
Bugs are often easy to find at the beginning of a test session, and they get more difficult later on until finally you are doing everything you can but not learning anything new or important about that part of the product.
Sometimes, this means it is time to move on. Be careful, though; other times, this is a sign that your approach isn’t paying off, and it is just time to try some new ones.
My manager said I should stop
Just like with writing code, testing can be a game of managing interruptions and context switching. I might be in the middle of testing a new code change when my manager gets a message that support needs help. They have a call from a high-profile customer about a data persistence problem. My new task is to get an export of that customer database, build an environment on the version of the product they are using, and configure it so that I can reproduce the problem as accurately as possible.
That manager interruption may be permanent, meaning someone else might take my original task while I work on the support issue. Or it may be temporary, and I return to the testing work once my role in the support issue is complete.
I don’t know what to do next
Sometimes, I get a code change and do everything I can, and then I start to question the mission. I have tested the text fields and I understand what happens when I enter unexpected data; I know that the date fields only accept one format and in a particular date range. But what now?
Not knowing what to do next is a good heuristic for a temporary pause. It gives me time to talk with developers to better understand their concerns, with product managers to get a clear idea of who the customer is and what they value, or with other testers I’m working with to generate more fruitful test ideas. If nothing comes from all of that, maybe it is time to move on to the next task.
The software isn’t ready for testing yet
Every once in a while, I get a product that either is dead on arrival or has so many problems that testing is like stumbling across a gravel beach in sandals. Sometimes, in these cases, I will make a decision to pause testing temporarily and return that feature to the developer so they can make some fixes.
This is strictly a matter of efficiency when there are easy-to-find, low-hanging-fruit types of bugs like forms that don’t submit or adding a decimal value to a discount that throws a JavaScript error. Each bug discovered represents the time consumed. That bug has to be investigated, fixed with a code change, merged into a new build, and tested again. When I decide to stop testing here, it is because the developer can quickly and efficiently find and fix these bugs.
Use your judgment
Earlier, I defined the word heuristic to mean a rule of thumb, but that also means it’s a rule that doesn’t always work. We might stop testing for anyone (or a few) of the reasons here, but there are also scenarios where we can break through and keep going.
For example, if you run out of time in the regular schedule but are still finding important problems in the software, your manager might be OK with giving you some more time. The more reasons you encounter — I’ve run out of time, and I’m not finding any more bugs, and management said to stop — the clearer it is that it’s probably time to stop testing.
Building a test approach
Knowing that there is an infinite amount of testing to be done, and also that it probably has to end soon, we have to select our test techniques and approaches carefully. It is hard to give specific ideas about where to take your approach, but I can point to two areas where people often get stuck.
1. Too much focus on things that aren’t testing: The role of a tester is filled with a lot of things that don’t involve interacting with new software. There is an assortment of meetings, emails, documentation, waiting for the next build, and sometimes writing code. Teams that aren’t careful sometimes find themselves crossing a threshold of spending more time not testing than testing. Sometimes these activities are helpful and teach something new about the software, but sometimes they are done only because “we have always done that.” Weeding out the aspects of your process that aren’t helping you learn about the software you are testing can be a good exercise.
2. Leaning almost exclusively on one technique: Domain testing is arguably the most popular test technique in the world. We use this strategy to analyze variables — text fields on a webpage, for example — and decide which values we need to test and which values might not uncover anything interesting. It is easy to get in the habit of entering some data into a field, observing what happens, and then declaring testing is done. There are a wide variety of test techniques out there, each designed to teach specific lessons about your software. Get friendly with them and learn how to use a technique and how it might be useful at a particular point in time. This can help you become a more effective tester in the small amounts of time that we usually get.
There is always too much testing to be done and never enough time to do it all. Having some ideas about what it means to be done can be helpful. Pair that knowledge with making your testing more powerful in the time that is available, and you will make the most of your testing efforts.
