This is a guest post by Peter G Walen.
Test teams are under ever-increasing pressure to deliver better software and increase depth, coverage, and quality of testing in shorter amounts of time and somehow figure out how to automate our tests while we are figuring out what testing is needed. Test managers, leads, and practitioners often turn to the process of testing itself to gain these improvements. Even when testing is “good,” the question becomes, “Is it good enough?” Is that the right question?
When it comes to process improvement for nearly anything, I try and apply the same approach: There may be a problem; Instead of whacking the symptoms individually, look to see what the symptoms are as a whole and see if there is a way to isolate the problem.
When looking at Test Process Improvement, the problem that gets described is usually a symptom or list of symptoms – not the actual problem. We can stomp on symptoms one at a time without addressing the crux of the problem. That will continue to churn and bubble and fester until something else breaks.
The “problems” are usually presented in a handful of ways:
- Testing is taking too long;
- Testing costs too much;
- Too many defects are being found late in development or outside the test environment.
Of course, there are other variations, but what I have seen have usually falls into one or some combination of those complaints.
One easy reaction to each of those points might be “Compared to what?”
A more balanced response might be, “Can you explain what you mean by ‘too long’, ‘too much’, or ‘too late’ so I can understand better what you expect?” Much of the time, estimates on test effort or duration are based on the understanding of what the project will entail early in the project, when little is known. As work progresses, we learn more and that will impact both the revised effort estimates and the actual duration. What we are doing is based on the instructions given to us and the mandates for what critical processes must be validated. Perhaps this needs to be reconsidered.”
So, I guess that’s a long-winded version of “Compared to what?” but it tends to go over a bit better.
What I do wonder, however, when a boss-type says something like those statements, is “Are those symptoms or problems?” Are we running over timelines and cost estimates because of other things than lousy estimates? Are “due dates” being missed because of testing? Is there a flurry of defects keeping customers from using the new release?
Is there something else going on and these are perceptions of problems, rather than symptoms of problems?
Self-Examination
There are a couple of things that most people find a challenge. One is to change perspective and see what others see in us. The other thing is harder by far. That is to look honestly at ourselves and see what we are really like and doing.
Any time that self-examination comes into play, most folks will avoid it as much as possible. We can tell wee little fibs and stories and justify actions by many interesting machinations. Yet when we strip those aside, what are we left with?
That is the hard part of any form of process improvement. When we look at ourselves, as individuals or as a group, the challenge is to set aside what we wish we were doing or like to think we do and focus on our true actions.
If our real processes and the official process don’t match, we need to ask ourselves, “Why not?”
Indeed, to see ourselves as we are, or as others see us, can be an event that we don’t want to face. Without doing that, we can never improve.
Process
There is a common assertion around software and testing that “If you don’t have a Process, you need one. If you don’t have a Process, you have Chaos and Chaos is bad.” The only problem is, most of the time people use the word “Process” when they mean “Process Model” – the official, defined path everyone is to follow. (For simplicity, I’ll use Process to mean ‘process model’.)
Having a process is not a silver bullet. Simply having a process will not magically fix a chaotic environment. If you are trying to impose a process on the organization wearing your “Tester” white hat or the plate mail of the Quality Paladin, good luck. Most places where I’ve seen chaos rule, it is because a very senior or influential person likes it that way.
Yet, if you have a process and no one follows it, the question should be why not?
When you look long and hard at what you and your group are doing, when you find the places where what is done varies from what The Process says, you must determine why this difference exists.
I suspect that it will boil down to a matter of relevance. The official process likely has no relevance in the context of what needs to be done. If it is a one-off, then maybe you can tweak it. If it is a regular occurrence, then the value of The Process comes into question. If it doesn’t work, why pretend it does? Why bother having it at all?
Granted, The Process may have been relevant at one time and things may have changed since it was introduced. Still, nothing is permanent. Change is inevitable. Even The Process needs updating from time to time.
Purpose, Mission and Charter
When you do, look to the Purpose your team is to fulfill. Why do you exist? What is your Charter? What is your Mission? Do you have a mission?
I’ll bet you do, even if you don’t know what it is.
To start, look at what management expects. If a boss-type is telling you that the test process needs improvement, try talking with them. Discuss with them what they believe needs to be improved or where the gaps are. This may become the basis of the group’s charter.
What are they seeing as “broken” that needs to be fixed?
If the complaint is “there are too many defects being found by customers,” ask if there are specific examples. Anecdotal evidence can paint a compelling story, yet without specific examples, you may never be able to find hard facts. Is this a hunch or are there specific examples? Are these defects that should have been legitimately found in testing, or are these the result of customers doing something completely unexpected?
It is possible these are aspects of the application customers expected to behave differently than what the intent was? If they are, why is that? What led to such a failure in communication and expectation management for such a difference to exist? After all, the design and requirements that you based the tests on matched perfectly!
Test Process Improvement
Each of these things plays a part in the entirety of Test Process Improvement.
Test Process Improvement (TPI) is often held up as an end in itself. However, TPI is not the point. TPI is not the end goal. TPI doesn’t matter except as a step toward the real goal: better value from your testing effort.
Here’s why.
Most TPI efforts focus on a linear, or sometimes a circular, self-referencing process model. The problem is that most humans don’t think in a linear fashion. I know I do not, particularly true when I’m working on a problem.
If I did, I would long ago have stopped looking into something I was testing because it did not feel “right”, even though there was nothing on the surface to indicate there was a problem. When I have one of those feelings, I sometimes will go over what I have for notes, look at the logs from the applications (not the nicely formatted versions, but the raw logs) or poke around in the database. Sometimes it is nothing. Sometimes, it is something, and often, the result is a very big something.
That is the pay-off for me as a tester. I found something fairly hidden with a strong likelihood of causing grief for the users or customers which will in turn cause grief for my company.
None of that is something that can be described in a linear fashion. I certainly don’t know how to describe it that way. I wish I did, I’d probably be able to make a pile of money from it and live comfortably for the rest of my life from the earnings.
The fact is, it is too organic, in the sense that Jerry Weinberg used the term “organic” the first time I encountered it in this context: Becoming a Technical Leader.
The Rub
The Test Script (and its companion, the formal Test Process Document) is not the Test. The Test is the part that is done by the M1-A1 Human Brain. Using that most powerful tool is the key to gaining value from testing — or improving the value you are currently getting.
You can have the best process in the world of software. You can have the best charter and mission statements. You can have the best tools money can buy.
Without encouraging your people to think when they are working — and rewarding them when they do it creatively and do it well, none of those other things matter.
Peter G. Walen has over 25 years of experience in software development, testing, and agile practices. He works hard to help teams understand how their software works and interacts with other software and the people using it. He is a member of the Agile Alliance, the Scrum Alliance and the American Society for Quality (ASQ) and an active participant in software meetups and frequent conference speaker.
