This is a guest posting by Johanna Rothman.
Agile approaches depend on pervasive testing. However, many senior managers think they can eliminate the entire testing department. Learn how to make your role as a test manager even more relevant in an agile organization.
Audra, the QA Director, walked back to her office, dazed. Her new boss, Cliff, the CIO, was ready to eliminate the entire test department. His reason? “We’re using agile. The developers do enough testing.”
Audra had negotiated a three-day reprieve before he eliminated the entire department. She was going to prepare data–real data, he said–to show that the QA department was worthwhile.
Cliff had quoted Peter Drucker in their conversation:
If we agree with Drucker, managers might then care about:
- The cost to acquire that customer. That’s the cost and value of testing.
- The time it takes to acquire that customer. One way is to talk about the speed of delivery of a given feature.
- Customer experience. Customer experience is a function of the usefulness and ease of the feature, the quality attributes.
Let’s take each of these in turn.
Assess the Test Costs and Value
Many people think that testing costs time and money. It does. If you only think about the costs, you might not realize the value that testing offers.
Last month, Audra’s team found a security problem that the developers missed. In fact, the testers only discovered the security hole when the developers fixed a different problem. That original fix exposed this new security problem.
In the previous iteration, one of the testers had discussed potential risks with the product owner and the security developers. The security team didn’t think they would expose that risk. The tester disagreed, and they all agreed he could develop some tests to check for that particular risk.
Everyone was quite surprised when the risk occurred. The product owner said, “We would have put all our customers’ data at risk. I’m not even sure how to assess the losses if that had gotten through. It shouldn’t have gotten through, and the tester discovered it. We spent two days developing a variety of tests that saved us millions of potential liability.”
Audra knew she couldn’t depend on just the huge problems to show the value of testing. She took the most recent five stories, one from each team, and did a cost/value analysis. Aside from the security problem, the testers had helped the developers think about the features in innovative ways and to decrease total lead time: the time from when the team took the feature onto their board to when they released the feature to a customer.
On average, testers helped reduce lead time by anywhere from several hours to one day, and on average, the testers reduced relative risk by at least half.
Audra expressed that data in dollars and continued.
Assess the Speed of Delivery
Too often, people assume that testing takes “extra” time and that developers would be faster without testing. Cliff certainly assumed that.
However, Audra had asked testers to affiliate with specific teams. The testers and developers learned to work with each other, for months at a time. The testers were an integral part of the team. They discussed the stories, the architecture, how certain tests would work to guide the design. The teams took the idea of “Card, Conversation, Confirmation” seriously.
Audra asked the teams to report just one example from each team where the testers’ contributions to the conversations helped the entire team release faster.
She discovered data that ranged from several hours to–in one case–two weeks. In that case, the tester realized that several paths weren’t necessary and could be safely eliminated.
We most often think about the product owner pruning the stories. In this case, part of the “testing” conversation helped shift everyone’s minds.
Audra gathered that information in dollars, too.
Assess the Quality Attributes
Customer experience is often a range of experiences. Audra decided to focus on the quality attributes of the software: usability, performance, and reliability. She asked the teams to explain just one story in the last few weeks where the testers had affected the usability, performance or reliability.
She heard stories about the testers had helped redesign a workflow that the customers preferred. She heard a number of stories about how the testers helped create scenarios for the developers to measure performance and reliability.
She gathered this data, and converted it to dollars. Time for another conversation with Cliff.
Explain the Data
Audra showed Cliff the data. She explained how much money the testers had saved with all their ideas and actions.
Cliff was still skeptical. “If I call someone, will they agree with your conclusions?” he asked.
“Yup, you bet,” she said. “Go ahead, call anyone.”
Cliff first called the product owner whose team had discovered the security problem. The product owner raved about the testers.
He decided to change tactics. Cliff next called one of the development directors. He asked about the value of the testers. The director said, “Not only does our team work well together, but the testers also help teach the developers how to test. They make suggestions for unit testing. When we do code reviews, they’re part of the review team. I don’t see how we would work without them.”
Cliff leaned back and said, “Well, I guess you and your people do add value. You’re not just a cost center.”
Audra said, “Nope, the testers here are part of integrated agile teams. We all work together to create the best possible product in the least possible time.”
Capture Data As You Proceed
Audra never expected this kind of conversation. And, when teams work as cross-functional, collaborative agile teams, you might find it difficult to collect this kind of data.
It might be worth your time to collect the data, especially when teams encounter problems. How do the developers help the team? How do the testers help? How do the other people enable the entire team to finish, and finish well? That’s the kind of data management might need to hear.
