This is a guest post by Rachel Kibler.
We have too many regression tests.
We have more regression tests than can be run in a reasonable amount of time by a small number of people. I’m sure this is a common complaint.
At my company, Anonyome Labs, one of our products is MySudo, a mobile app for iOS. It does a huge number of things, and the amount of stuff that can break feels astronomical. We run about two-thirds of our entire suite for regression testing of our monthly release. If two people were working on it alone, all day, disregarding new work, it would take about two weeks. This is an untenable situation, so we use TestRail to facilitate company-wide testing.
And we do have representatives from the whole company who help: testers from other products, development, user support, design, marketing, product, tech-ops, executive staff — even our accountant ran five test cases last time. They all have their different experiences and different ways of using MySudo, and the combination of running scripted test cases as well as just using the app in their daily lives means that we end up finding problems (mostly) before our external users do.
Regardless of my feelings about scripted testing for myself, it becomes very useful when engaging non-testers to run regression tests. I can assign 20 test cases to a marketing person, and they know exactly what to do, complete with screenshots. They can easily mark a test as passed or failed, or they can send it to me with questions.
A normal regression test cycle involves roughly 10 non-testers and five or six testers. Because of all the extra help, instead of two weeks to run regression testing, we get it done in roughly three days.
As test lead, I take the tests that require special access or are just a pain to run, and then I review all the failed test cases and ones marked for retesting. To have a hygienic run, I confirm all the failures and make sure the appropriate bugs are attached. The testers who are committed just to this product help with this, and they jump in if someone gets swamped with their normal work and cannot run their assigned test cases.
Using TestRail and having the whole company test alleviates the pressure on any single person … except me. While I don’t have to do as much testing myself, I end up herding cats, fielding questions at all hours (we have offices on multiple continents), and doing the hard stuff myself. It’s mentally and emotionally draining, but the benefits outweigh this cost.
Test cases can become stale, particularly if the same person is looking at the test cases over and over, and having fresh eyes on the test cases as they are run allows us to pick up on things that may have been missed before. We can mark tests as needing updating, or we can update them on the fly.
Part of what I love about my job is that everyone believes in what we’re doing, and we all pull in the same direction. Having the scripted test repository means that more people can help us achieve our end goal faster. People in the company seem to enjoy it, and they are engaged and committed to what we do.
I had only used two other test case repo tools before TestRail, but this one seems uniquely positioned to facilitate the type of testing and amount of external engagement that we regularly have. And now that I’ve done it this way, I don’t ever want to go back.
Rachel Kibler is a test lead at Anonyome Labs, which makes MySudo. She can be found on Twitter @racheljoi or on her website at racheljoi.com.
