This is a guest post by Peter G. Walen.
Regression testing can be a pain. It can be slow and cumbersome. It can be problematic. It can be a pain to figure out and to get right. And we haven’t even touched on how to do this with automated tools or in an iterative environment.
In my 30+ years of working in software, I’ve heard scores of definitions of what “regression testing” is. Just to be clear, here’s my working definition of Regression Testing:
Regression testing is the testing work done to make sure that functions which are central to customers being able to use the software, which worked fine three builds/releases/iterations ago still work fine.
That’s it. That is how I approach the idea of regression testing. There are solid, academic definitions. I tend to avoid them as I have not worked at a University since the mid-1990s. I have found this to hold up pretty well for many years.
How do I figure out what should be included in a regression test or a collection of regression tests? The simple answer is, “Not everything.”
I can recall very few instances where every aspect of every release warranted being included in a regression test suite. I tend to look at this in a similar way as I do test automation – do what makes sense.
If the functional test used for a given build or release exercises an area that will be a critical success factor for the entire product, it certainly is a candidate. If a function exercises a component that is not terribly important in itself but is used by important components in other parts of the software product, it certainly is a candidate.
It is not a guarantee these will be included in a regression suite, but they will get looked at more closely than a minor change of some sort.
Not everything
I’ve given a couple broad ideas on what might be included in a regression test suite. Here’s what I mean by “not everything.”
I’ve seen organizations look at taking functional tests and port them directly to the regression suite. This might make sense, in a few limited circumstances. Where the logic falls apart is when there is a change to the same part of the software, a minor tweak to functionality, and the new functional tests are included along with the original tests.
Then there is the case where there are regular changes to the same piece of functionality. Does it make sense to include something which will be changing every iteration in a regression test suite? It might in the right circumstance. I’m not sure I have seen it.
What I have seen, and done, is to make a note of important and vulnerable areas being worked on. As they get closer to “ready” with fewer changes coming, I start building a basic set of checks. We don’t need to exercise every single combination of possibilities. That has already been covered in functional testing.
Focus on the main point of the function. The minutiae of changes and possible flows are less significant than they might appear.
Is this a safe approach? It doesn’t sound very safe, does it?
Consider this, if the feature is changing every iteration, does it make sense to change the regression suite code for these minor changes? If they are likely to get exercised by the functional testers in the next iteration’s testing, is it worth the time and effort to change the regression test?
More not everything
During a detailed functional test, there may be a need to verify the contents of a dropdown list. Actually, there often is a need to verify such a list. It makes perfect sense when there are oh, a few elements. When there are over 15 or 20, I suspect most people don’t really pay too close attention to the actual contents.
When the drop down list gets over 100? What? There may be some folks who go through line by line verifying the content. Most will, I suspect, compare the contents of the dropdown list with the official source. If there are the same number of entries, they declare it “good.”
Don’t get me going on the drop downs that consist of 1100+ entries. (It is a thing I’ve seen.)
When doing regression testing, should any of those tests “make the cut” to be included in the regression test effort? Not likely. If there is a drop down to select a value from, select it and move on. Usually, dropdowns like this are pulling from a DB table. When I have seen these vary in content it has been the result of environmental issues or, in some cases, someone messing with data because “it is only a test environment.”
Focus on what is important to the function of the software. The small, technical details should be exercised during functional testing. Don’t spend time designing tests to exercise these when you should be looking at the main usefulness of the software.
If you see a dropdown with 3 values when it previously had 150, it might be fair to ask about it. Putting in the time and effort to exercise every possible dropdown? That probably does not deliver the same value.
But what about…
Yes. There will likely be some things that are really important to the product owner or product manager or someone overseeing the business function of the work. These need to be discussed and evaluated. Explain the amount of effort needed to do this manually each and every time. Explain the effort needed to write and test the automated scripts that would test this.
By all means, talk about the things that need to be done anyway in order to test new features in function testing. Are these part of the consideration? Make sure this is included in the conversation.
If the cost of doing these things for regression testing is offset by the business confidence in the feature being tested then it may make sense to include them. The important thing here is to discuss the effort and the associated cost of doing these things. Is the effort, time, and cost for features with no changes to be exercised again and again, greater than the value of adding other features to the regression effort?
And what about…
There likely will be features and functions that are core to the needs of customers, the people actually using the software. These also need to be discussed and evaluated. This also brings us closer to the main goal of “regression testing.”
Testing features or functions from previous builds is important and makes a great deal of sense. What gets missed by many organizations is the question of “does the software continue to meet business needs?”
We want to believe it does. To achieve any level of certainty, we need to remain focused on the reason behind the software we are working on. What do the pieces, the features and functions, all add up to?
The way I normally structure formal “Regression” testing addresses this in many contexts. I look at the logical flow expected by the people who need to use the software to do their work. Then scripts can be built around these flows which include the variations they see frequently.
These scripts can be exercised by people working through them. They can be scripted and adapted for a test automation platform. Each of these flows is evaluated and built to address the needs of people using the software and the business problem the software changes are intended to address.
By doing this, you can build a meaningful collection of regression tests that can be adapted and changed as the software changes. Not all changes are equal. Focus regression tests on the most important business flow for the customers. Wondering how to use TestRail for your regression testing? Check out this blog post!
