This is a guest post by Peter G. Walen.
When we are working on testing a piece of software, one huge challenge is to note everything that might be important. Of course, part of the issue is, we often don’t know what might be important, and we often cannot differentiate that from what is not important.
When executing scripts developed by someone else, people are often told to focus on the “expected results” in each step or action. That is exactly what most will do. Attention gets focused on the very narrow result set. That is what the instructions require.
For many, doing anything other than that might lead to criticism or reprimand. They likely would be told to follow the steps precisely. We are not to vary or introduce any deviations, because variation is bad. Doing things not instructed to do, in the way they are directed, is equally bad.
Of course, if a piece of software tested in such a manner fails in production, there is no end of the reprimands and recriminations. Blame is heaped upon the tester for not finding the problem during testing. Blame is heaped upon the “testing team” for such sloppy and unprofessional work.
Never mind that the scripts they must follow with absolute precision never included the steps needed to detect the failure. They never suggested the conditions required to uncover the problem. Following the scripts as written, they had no way to detect failure.
Why the narrow focus?
Many organizations have directed people to do “testing” in that precise manner. Now they are doing “automated testing” in the same manner. They are focusing efforts on paths that are predictable to measure and estimate, amount of work as well as the time for each test.
This approach has resulted in several things.
One is a continued and sustained view of “anyone can test” and “testing is easy.” Another is the lack of need for technical skills among most testers – from reading DB queries in SQL to reading and understanding the code driving the customer-facing software. It also maintains testing as a different function from developing software and drives “testing” into something other than knowledge work.
These combine into a pernicious attitude that testers do not contribute to the organization and can be easily replaced by tools or automation, or simply have the developers execute tests because testers are not really needed.
How has this come about?
These practices strip people of one of the most important traits needed to do good software testing: Curiosity.
Even when something unusual is noticed, people doing the testing have been trained to ignore it. At the most, they might make a note in the “actual results” and move on. This makes perfect sense if someone reviews the completed scripts for such notes. Most of the organizations I have seen simply do not do this.
When they do take time to investigate the behavior, they have learned that they get reprimanded for “wasting time” if nothing comes of it. Even if a problem IS detected, the first question often asked is something along the lines of “Was this part of the test?”
The argument is “other teams” are running other tests. That may be. What are the odds these “other teams” would trip over the same conditions and situation which resulted in the anomaly? This also raises the question of why are multiple teams running detailed scripts against the same piece of software?
Many companies have trained people to ignore what they see. Others have failed in another way. They have not encouraged people testing to investigate unexpected behavior or results. This carries over to other aspects, teams, and for the people doing the work, future jobs.
We, as a trade, have barely trained people to “see.” We have not trained them to “observe.”
You see . . .
The word “see” is an interesting one. Common definitions run something like this: “to perceive by the eye; to perceive or detect as if by sight.” Of course, there are other definitions. I suspect most people will take the first idea and move on.
This is rather how many people get trained to “do testing.” Grab the obvious idea or result, and move on to the next step. Still, by doing so, we might miss possible definitions such as “to be aware of.”
Again, that might be similar enough to the first result to seem negligible. On the other hand, it might be the problem we find in how some, many based on my experience, organizations treat testing.
People may SEE something unusual. If they are not told to look for “anything that seems unusual or out of place” in addition to the “expected results” some, perhaps many, may be reluctant to report it. They do not want to be criticized for wasting time or doing “extra work” that is not part of the test.
As mentioned before, they may make a note in the “actual results” bucket, but if no one reviews the detailed results they add nothing for the project.
We find ourselves in a stalemate if you will. People have been trained to do specific things and not be curious if they find something unusual. They might get told, verbally, once, to look into anything that seems unusual. If they are not familiar with the application, I suspect they might not know if the behavior they are seeing is unusual or “normal.”
After a couple instances of reporting “I saw something I don’t understand” they might get reluctant to repeat the exercise. They might shrug their shoulders and move to the next task.
But do not observe
“Observe” is another interesting word. The Oxford English Dictionary gives us a definition of “notice or perceive (something) and register it as being significant.” This goes beyond “seeing” something, or noting it is present. It is one step past that. It recognizes there is some significance to it.
It is an action taken where there is some thought or consideration around what is seen.
The curious thing is, I have some exercises I use in workshops and training sessions intended to help people learn to pick up on cues and clues. These are a series of pattern identification and recognition exercises. In short, they help key people into the idea of “observe behavior,” in software or anywhere else.
The idea I try to convey to participants is that sometimes the biggest problems can be lurking behind behaviors that seem trivial. A small flash on the screen or a message that appears and is gone might be a clue to something happening elsewhere that is not so obvious.
Sometimes it is the wording in a message on the screen, a slight difference from the expected messages, which might be overlooked but actually give warning to other problems. Realizing the significance might take time. It often takes a bit of practice.
When testing, we can be so focused on what we are “supposed” to look for that we do not recognize the significance of other things. Even when they are right in front of us.
I drew a significant lesson on that from a fictional character, created in the late 1800’s. Sherlock Holmes, created by Sir Arthur Conan Doyle, spoke to this directly.
“You see, but do not observe. The distinction is clear.”
Holmes on testing
Consider another idea Doyle presented, through Holmes. “The world is full of obvious things which nobody by any chance ever observes.”
This idea, with the previous quote, combines to describe what appears to be the central issue around inadequate testing. People may see something but not understand the significance or value of it. Some things are readily available, not hidden in any way, and somehow no one at all recognizes them for what they are.
Not all bugs or problems in software are the result of poor coding. Not all bugs or problems in software are easily understandable. However, when looking actively, when observing the software in use, we might be able to comprehend the issues that have been before us the entire time.
The odd error message that pops up from time to time acts a bit like the Hound of the Baskervilles. That it appears is not the curious thing. When it does not appear might be the clue to the problem.
It is sometimes a challenge to focus on the task at hand, while being aware and noting what is going on around us. That is a challenge faced by more than software testers. Indeed, the phrase “situational awareness” was coined to help direct attention to that challenge.
We must be aware of what surrounds us. We must look beyond the simple surface presented to see the underlying behavior and activity. We must see how things are impacted by our test actions even when we are not “instructed” to by the script we are following.
Great problems may be revealed by the smallest clues, “trifles” as Holmes would describe them. If we study and learn the behavior of the software we are testing, we can learn to detect those which are significant from those which are interesting, but distractions. When testing, we must take in as much as we can, filter it appropriately, and present meaningful analysis.
The lessons
These are lessons often learned by hard experience. By learning to look beyond the scripted expectations, testers can begin to observe behaviors others, including those who wrote and developed the scripts, might not find. They will begin to apply the lessons from Sherlock Holmes when they observe and not merely see.
Peter G. Walen has over 25 years of experience in software development, testing, and agile practices. He works hard to help teams understand how their software works and interacts with other software and the people using it. He is a member of the Agile Alliance, the Scrum Alliance and the American Society for Quality (ASQ) and an active participant in software meetups and frequent conference speaker.
