This is a guest post by Rachel Kibler.
This may come as a shock, but we don’t release perfect software into production. Our mobile app sometimes crashes, some types of data don’t come through, and the navigation flows can be complex.
We find and fix a lot of bugs, including most of the regressions before they are released, but we still have a significant backlog of bugs that we want to fix at some point.
As the test lead on our flagship product, I live in the backlog, and I breathe these bugs. A normal release for us fixes roughly 20 bugs that are unrelated to new work. With a backlog of about 200 bugs to choose from, I had to learn how to prioritize bugs and select the “right” ones for releases.
I have no secret formula, but I do have a bunch of things I’ve learned. Here are five important tips for choosing which bugs to address.
1. Have rules
For my product, I run a weekly bug triage meeting with the product owner and developer lead. We look at bugs that have been filed, starting with the highest-priority ones. We evaluate the bugs, discuss the user experience, and change the severity, likelihood, and priority if we feel they need to be changed.
Instead of going with a gut feeling for priority, we have guidelines for severity and likelihood, and that gives us a starting place for a priority through a matrix we developed. The priority can change based on business needs, but it’s a useful piece of information when the matrix’s determination differs from the opinion of the tester who found the bug.
We all have our pet bugs, and we all think the bad experiences we find are of the highest priority. Having rules to fall back on gives us some perspective.
2. Make it a group effort
That bug triage meeting can be exhausting. The triage didn’t start at the beginning of the project, and I still have bugs from 2018 that haven’t been triaged yet. And though I know these bugs so well, I don’t triage them on my own. It’s a group effort, and it has to be a group effort.
We all have different experiences and different opinions. Going through triage together means that we are more thoughtful about the bugs and we bring clarity to things that were fuzzy before, and, honestly, it feels better to close bugs as “won’t fix” after we agree.
Another place I’m not alone is feedback. We dogfood our software, and our user support people sit right next to me. I can tell when a person walks into the room that they have a bug to show me by the level of eye contact they make with me and the way they’re holding their phone. Thankfully, most people are just impressed by their own level of ingenuity in finding bugs, and they want to be helpful.
3. Determine urgency
Some things have to be fixed right away, and they often take the form of hot fixes. Crises such as a huge surge in customer support calls or a sudden drop in an app store rating can make a bug very urgent. When the business is on the line, it is urgent. When it is something that comes in from our board of directors, it is less urgent but more important.
If urgent bugs can wait for a planned release, they do, and if they can wait for a release after that, then maybe they aren’t terribly urgent after all.
4. Evaluate importance
If something is about the long-term goals of the company or the product, it is important. These often go in as features. My team considers whether a bug has an impact on the “tranquility” of a user’s experience as well, and tranquility is a long-term goal of the company. Thus, fixing tranquility bugs often takes precedence over other types of bugs.
Maybe this point belongs under the first tip, “Have rules,” because really, it’s about knowing what drives the company and what the company wants to be in the future. What’s important will look different to each company.
The gradual decline of a lesser-used feature can be important until it becomes urgent if not addressed. Important bugs need to be addressed so that they don’t become a matter of urgency in the end.
5. Let go (a little)
I reserve a label in Jira: my “QA-list” label. I am the only one who can add or remove it from a bug. These are the bugs that, after triage and evaluation and conversation, I want to be fixed over other bugs. These are often the important bugs rather than the urgent bugs.
When we’re looking for bugs to fix in a release, we pull from my list. And when a dev finds some extra time, they know they can pick from my list. I try to keep it to 20 bugs that are not planned for releases, so the list is manageable and consumable. I set my parameters and then let go of control within those boundaries.
These things have worked for me and my team. I needed a way to get more perspective on a wider range of bugs, and understanding what was important and having conversations around that aspect helped create the process.
