This is a guest posting by Justin Rohrman.
Agile comes as a package deal for most companies right now. That deal includes Scrum, sprints, a kanban board (but probably not a kanban process), and acceptance criteria.
Acceptance criteria are supposed to represent the heart of a feature — the value a customer needs for the change to be useful. Product managers use acceptance criteria as a way to communicate ideas that will be translated into software, developers use them to make sure they are implementing the right thing, and testers sometimes use them as a testing outline. But, like any sort of software requirement, most acceptance criteria are incomplete, wrong, out of date, or misunderstood.
Here are a few ways testers use acceptance criteria, as well as some tips for how to make those criteria more useful.
Automation Design
Designing a good test to perform can be tough work, even when tools aren’t involved. A good test reveals important information about a piece of software as quickly and easily as possible. Mixing code into the equation can make the “important information” part of this tricky. I like to use acceptance criteria as a focusing tool for any sort of automated tests we build.
When they hear the word “automation,” most testers think of driving a browser, so let’s start there. There are unending combinations of tests we could write. For just one text field, we could try null, a short string, a long string, numbers, special characters, Unicode, and so on.
But automation is code, and code is a liability, especially in the case of browser automation, where tests can sometimes randomly fail. Code has to be carefully crafted so that it is maintainable, and in the future, it will most likely need to be revisited as the product evolves. We want to write the absolute minimum number of tests (and lines of code) that will alert us to when customer value is at risk.
I like to play reductionist when I build tests. Acceptance criteria might represent a workflow or a change to an existing workflow. I may reduce a set of acceptance criteria into one test if possible, especially if parts of the change are tested by unit tests that look specifically at the model, view, or controller.
The acceptance criteria aren’t a list of things that need to be automated, but they can guide me to test ideas that might make sense to automate.
A Dev Checklist
In my experience, this is one of the most useful places to use acceptance criteria. The feature changes I have worked on include layers of work: research, database changes, architectural decision, and design and user experience considerations. I like to use each area of criteria as guidance on what to do next.
On my current team, we start a change by looking at the acceptance criteria and asking, What is this thing supposed to do, and what do we need to do first to get there? After we decide what to work on first, we write a test that will fail because there is no implementation yet. Next, we write the implementation, and then go back and refactor code to make it more readable, maintainable, or performant. We do this in cycles until we can check off each item of acceptance criteria on the list.
The acceptance criteria checklist probably doesn’t encompass everything a person might expect from this change. Testing is a part of writing code, so while we are making these changes, we have to run the product and ask how it will actually be used. Does this workflow make sense in the context of the rest of the product? Will the thing we built actually satisfy a need? Are we missing something important?
A Riff on Test Cases
Exploration is the basis of many testers’ work. I can come up with, and run many test ideas very quickly, and usually learn something interesting about the product in the process
A shallow way for testers to use acceptance criteria is to make each one a “test case,” perform those tests, and then dust their hands together and say they’re done. This was a common technique when heavy software specifications were in use more often.
I like to use each criterion as a riff — or as an improvisation if you’re more into jazz. Each piece of acceptance criteria is a theme I can build off of and change a little bit at a time. In testing terms, I would take one criterion and start thinking of test ideas I want to perform. Each time I perform a test or a series of tests, I’ll probably learn something that changes the order I do things or completely changes my plan.
Acceptance or Rejection Criteria
I occasionally see complaints that acceptance criteria are terrible in the same ways that old-school specifications are: They are incomplete, wrong, out of date, misinterpretations of what the customer wants, or misinterpreted by the people building the software. There is also the implicit part of acceptance criteria that states something like, “We want all of these things … and for nothing terrible to happen.”
Those same people will sometimes reframe acceptance criteria as “rejection criteria” — as in, if one of these things is missing, then we cannot ship this change to production. I find the concept of rejection criteria not very useful for that exact reason.
Rejection criteria can’t quite capture what a person might expect from a software product, and they certainly can’t capture all the different things that might go wrong. There is a largely hidden territory when you frame these criteria from any particular angle, so let’s treat acceptance criteria as what they are: imprecise guidance.
I get the most value out of these criteria during a Three Amigos meeting. Before we start implementing a change, a product person, a developer and a tester get together to talk about what they need to do. We walk through and analyze the acceptance criteria one at a time. Generally, we are looking for whether the request makes sense, how it integrates with the rest of the product or other projects that use the code base, how a person will want to use this software and how we might test the change.
Each criterion in the checklist is a prompt for questions that will help us refine our understanding of the change we are supposed to make. Sometimes these questions help us improve the request itself, and sometimes they help us reject the request if it doesn’t make sense in the context of our product.
Let Acceptance Criteria Be Your Guide
Acceptance criteria are useful to a lot of people in software in a lot of different ways. But they should be used as prompts, not as explicit instructions. The main thing to take away is those acceptance criteria are guidance along your path for building quality software.
