Tester’s Diary: Rethinking Test Reporting

With the project I started last year finally behind me and my new project not yet ready, I was able to take some time last week to work on my annual goals. I hoped to start by meeting with the product manager of my next project to discuss the basics of test reporting: what information she needs, what decisions the test reports drive and what is missing from our current reports.

My plans were foiled by some good news: The product manager was promoted to director. A new product manager was announced for my next project, but she was out of town when I had scheduled the meeting. Oh, well. I scheduled a new meeting with the new director, the project manager and the test manager, and I decided to continue researching test reporting styles and methods while I wait.

The one thing I know is that our old method of test progress reporting did not work. Our initial attempt at reporting counted how many test charters were written vs. how many test charters were completed. This approach had a number of flaws.

First, the number of test charters required for the entire project was unknown. I wasn’t writing charters for features or defects until they had been added to the backlog. This meant that the number of charters written wasn’t meaningful, because that number increased over time. Further, the charter count wasn’t a good metric regardless of how many there were, because I wasn’t using time-limited testing against charters. Some charters could take an hour, but some charters took a full day just to create the appropriate test data or evaluate the results. Consequently, it was very difficult to understand how much work had been done or how much work remained, much less what the overall quality of the product was.

Last year I tried alleviating this problem by creating a mind map. Because most of our test work is based on testing features and defects, I used our feature and defect-tracking software as the basis for the mind map. The program we were using at the time included a project tree, which we used to organize the product features and defects into epics. My goal was to create a script that would export that tree into a color-coded mind map so that it would be easy to see which features were tested and defects were found.

I exported the project tree as a CSV file, and I converted it into a mind map by hand-drawing using an online mind-mapping tool. This mind map was extremely useful to me because I could see not only which features were tested and which weren’t, but also that there were areas of development that weren’t fully fleshed out, because there were epics with no features associated with them. I could also see that there were some sections of the tree that had many more defects associated with them than others, so we could probably consider these features fragile or in need of more attention. This was an exciting result!

I showed this mind map to my test manager, and she was impressed as well. The test manager and I agreed that there was high value to this report, so we decided to show the product manager. However, the product manager didn’t seem to understand what she was looking at or how it was valuable. It seemed to her to be more of a development tool than a tool for making product decisions. This was not the reaction I was hoping for.

An Alternative Test-Reporting Format With a Simple, High-Level View

After doing some research, I’m looking at an alternative test-reporting format in the form of a dashboard. The dashboard is similar to the mind map in that it focuses on the product as a whole rather than individual features and defects. It differs from the mind map in that it offers a qualitative assessment only at the basic level. Where the mind map offered the ability to dig deeper into the project tree to assess the project, the dashboard would allow a simple, high-level view.

Having examined both formats, I believe the test manager and I could use the mind map to create and update the dashboard, allowing us the insight and detail we need to make testing decisions and inform development of potential problems, while giving the product manager and director only the limited information they need to make decisions. This is the proposal I will make at our scheduled meeting — after starting the discussion about test reporting and determining whether this tool would really be as good a fit as I think.

Finding ways to make our testing more visible and usable to the project stakeholders is really exciting to me, and it’s something our team needs. It’s amazing that such a simple change in how the testing team carries out their work can improve the performance of all the teams that work on the product. I’m glad I chose this as one of my annual goals.

This is a guest posting by Carol Brands. Carol is a Software Tester at DNV GL Software. Originally from New Orleans, she is now based in Oregon and has lived there for about 13 years. Carol is also a volunteer at the Association for Software Testing.