The Workshop on Teaching Test Design was held in Columbus, Ohio in February of 2015. The goal of the workshop was to come up with a set of exercises that can be freely distributed to help people teach test design. I grouped up with one other person, Paul Harju, and wrote a small web page in JavaScript. The webpage has a text field and a button. The user can enter a string into the field, click the submit button, and then in the the website will refresh and tell you whether or not the string you entered is a palindrome. This looks like a simple challenge at first, but there are a lot of different directions in which the exercise may be taken.
Get TestRail FREE for 30 days!
Planning
Planning is something that sometimes get skipped in the age of exploratory everything, agile, and delivering multiple times a day. I mean planning in the sense of thinking about what of the goal of your testing should be, not creating documents to put in a wiki and never be looked at again. A plan guides our testing work and reminds us of the current mission, and that is where I like to begin with the Palindrome Testing Challenge.
To build a plan, you have to know something about the software you will be testing — why someone would want to use it, who that person is, what they value, and so on. A more advanced tester will ask questions that lead them toward customer value, so I like to have a story prepared to answer questions with. When asked, I like to tell the person taking the challenge that this software is designed to be used for elementary school teachers to teach the concept of a palindrome. The software is completely free and available over the web. The story allows me to play the role of product manager and opens up a line of inquiry about what the teacher needs from the product, what the student needs from the product, and how they plan on accessing the software (browser, operating system, mobile or desktop, etc).
Less skilled testers might talk about planning by giving you a list of test ideas. These testers will tell you about all of the different strings they will enter into the field, how they will submit those strings, how they will try to overwhelm the software and crash a browser, and the different browsers they will use. I did this exact thing in my first testing challenge. I created a stream of consciousness list of tests that would thoroughly test the product we were talking about. This pattern shows that the testers mind is on and ready to learn.
Building a plan for this exercise should only take a few minutes. It should help us to think about what testing we want to perform when we know very little about the product, and enable us to learn quickly.
Testing
Testing is the fun part of the exercise for me. There are some bugs in the Palindrome Calculator; some intentional and some from mistakes in the JavaScript, so you won’t walk away from the testing wondering if anything you did was useful. You or the person you are navigating the exercise with should already have a plan, so you know what tests you want to perform or at least where you want to begin. When running the exercise myself, I’d give the person 10 or 15 minutes to test the product and see what they come up with.
Outcomes will vary depending on the person doing the testing. Some people that have come from a more rigorous or regulated testing environment will focus very heavily on the plan and lose time by focusing too much on what they prepared ahead of time. Some testers will find a bug and spend a large amount of time researching the bug, the different ways it might manifest, and the different ways to trigger the behavior while time is running out. Others will start testing immediately and jot down small notes about things that are interesting that they might return to at a later time.
Each of these patterns tells you something about the tester, or maybe about yourself. Namely, this is a good illustration of the different things we do while testing software that take away from actual test time. It is hard to say whether such as referencing and sticking to a plan, researching bugs as fully as possible, and note taking help or hurt testing work without further analysis, but we should note that they occur so we can think about them.
While the person is testing, I like to ask questions to challenge their thinking. What percentage of the product would you say you have tested so far and how much is remaining? Tell me about the test you are performing right now, what problem in the software are you trying to discover and how did you come up with that test idea? Are you done testing, and if not how much more time do you need to be completely done?
The testing part of the exercise is a dance between the tester and the person running the exercise. The tester tries to discover important problems as quickly as they can and communicate those in a useful way while the person running the exercise adds pressure in the form of shrinking time and questions.
After testing the palindrome, you will want to talk about what happened. This is called a debrief.
Join 34,000 subscribers and receive carefully researched and popular article on software testing and QA. Top resources on becoming a better tester, learning new tools and building a team.
Reporting
This is where you debrief on the testing that you or someone else performed. There are a few main questions to cover here. What exactly did you test, and why? Was it good testing and how do you know? Are we ready to ship this new software to production? Discussing reporting can be an exercise on its own for some people.
Testing happens subconsciously for some people. A tester will go into a piece of software and navigate to the change they want to look at and just dive in. They might not think about where to start, or the best way to test the thing they are looking at, or even why they are testing, but they test nonetheless. Asking a person to recall what they tested and talk about those decisions is one way to help them think about the work they did, and what else they might do if they had more time. The second question about the quality of their testing work can be difficult even for skilled testers. They might talk about their work from the perspective of how much of the plan was covered and why they decided to add or remove parts of the plan. Or they might talk about the quality of their work based on the problems they discovered and how they would be important for the people using the product. A holistic answer would cover both of these. It is more important to consider what makes testing good or not so good than to have a solid answer about how good the testing was.
The last question is tricky. Some in the testing world feel that it isn’t the role of the tester to make decisions about when to ship new software. Their point is that testing is an information delivery service and that testers don’t have any control over project variables like the timeline, staffing, or the scope of each release. Despite those valid points, in my experience testers are often asked about their feelings on a particular release. I use this question as a tactic to add time pressure to the exercise, just like we experience in real software projects. Each time I ask if we are ready to release, the tester has to reconsider where they are, how much work they have done, and how much more they need to do do be able to provide a useful answer to the question.
This palindrome testing exercise is fun, only takes about an hour and you can find many themes in real software testing projects embedded in the practice. When I run the exercise, testers think about planning, test coverage, working with uncertainty and time pressure, and deciding what it means to be done. This isn’t the only way to run the exercise of course. You can focus on test design, or security, or usability, or planning. Why don’t you try running the challenge during a learning session at work or a local meetup, and let us know how you got on?
This is a guest posting by Justin Rohrman. Justin has been a professional software tester in various capacities since 2005. In his current role, Justin is a consulting software tester and writer working with Excelon Development. Outside of work, he is currently serving on the Association For Software Testing Board of Directors as President helping to facilitate and develop various projects.
