This is a guest post by Nishi Grover Garg.
How can we study bugs to improve the overall state of our project? Here are three things to learn from the bugs you find.
1. The location of defect clusters
Defect clustering is one of the seven principles of software testing, and keeping an eye out for these clusters is the responsibility of a good tester.
As we log defects into a tracking tool or portal, teams generally follow the practice of measuring relevant modules, components or functional areas against each defect. When tracked over time, this information can be real gold! It helps us track which areas of the application are having more bugs.
We can plot these area metrics against the number of defects raised and find the defect rates over time. We can also create filters to raise concerns whenever the defect rates go over a certain limit in any specific area or component. This can help us combat defect clustering by doing a fresh analysis, revisiting the tests being performed, and focusing our exploratory test efforts in those areas.
Overall, knowing about these defect clusters, keeping an eye out for them, and regularly revisiting the areas will help us keep the quality of the entire system in check.
2. The frequency of defects (and their resolution)
The frequency of defects being found and logged tells us a lot about the maturity of the product.
At the beginning of construction sprints, defects are supposed to be frequent and plentiful. We may not go by numbers here, but the relativity of them. As we progress toward a release, the number of defects generally declines, indicating that the system is now more mature and sturdier after withstanding multiple test cycles. Some teams even use the metric of mean time between failures as an exit criterion for testing, indicating that they will only finish testing once they cannot find any new defect for a certain number of days.
As defects are raised, triaged, resolved, and verified, there is a typical turnaround time that we expect. Most defects will go through this lifecycle within a reasonable stipulated time or will be postponed with a reason or business decision. Some defects may linger in the system for longer.
There may be a variety of reasons for these decisions:
- A defect requires more information, and the developer is awaiting confirmation or details from the tester who raised it
- The defect was misunderstood and there are comments going back and forth between the tester and developer about the expected behavior
- The assigned developer was on vacation for a week and the defects have not been fixed, leading to a plateau in the defect-fix-rate graph
- Defects are awaiting triage by the product owner and do not have priorities or the correct people assigned to them
Whatever the reason, knowing the cause of defects remaining open, in progress or unresolved for longer than a stipulated time is important. We may have to fix people issues or communication gaps, or may just need to schedule a short triage or discussion with the team to decide on the fate of such issues. But understanding any delays gives us much-needed insight into team dynamics and helps us smooth out the process.
3. The reasons behind rejected defects
The number and type of defects getting rejected — and the reasons behind the rejections — can also tell us a lot about the state of the product and the psychology of the team. If you see a high number of irreproducible defects, it may mean that some data or information is getting lost when reporting, or that the testers do not have enough time or perspective to reproduce the defects.
A high number of duplicate bugs may show that testers are unaware of the system’s history, or maybe they are new to the team and need to get a little more background. It may also be a case of the same bugs reoccurring, which might have been fixed and closed in previous releases.
Incorrect defects marked with “Not a bug” or “Working as designed” tell us about a lack of understanding of the system on the testers’ side. Or it may be due to a lack of communication among the team members, leading to different perceptions about the features that were designed or implemented.
Our findings from these types of defects can help test managers or project owners plan measures like internal training sessions and knowledge sharing, which can enhance communication among team members and introduce prerequisites to fulfill before logging any issues.
There is a world of information that your defects can provide. If you take a good look at your bugs and talk about them as a team, you can find ways to use that information to your advantage.
