Testing doesn’t make software better.
I get a container with new software and deploy that to a test server several times every day. I take relevant context — the code changes we are making, what has been tested programmatically, what hasn’t been tested programmatically, feelings about how good the developers I am working with are, and thoughts about the brittleness of the area we are working in — and then perform some testing. I design tests with all that in mind, hoping to find a new problem that matters. Sometimes, I strike gold.
Despite all of that, nothing I have done while testing actually changed the product quality. What I did do was learn some important things and share them with other people. Testing doesn’t affect product quality, but testers do. Here are seven things testers do to make a contribution to quality.
Get TestRail FREE for 30 days!
1. Designing Better Tests
I said earlier that finding bugs doesn’t affect product quality, but I’m about to contradict myself. Deal with it.
Good test design, or the ability to design an experiment and learn something about your product that other people will care about, is one of the more important aspects of testing. I can design good tests that find bugs, and maybe those bugs get fixed or maybe they don’t. But either way, sharing those test design skills makes a difference.
I work very closely with developers through the entire feature lifecycle. This starts with requirements and ends with pushing to production. Throughout that process, we are building tests. Some people call those “checks,” and that’s fine, too. My ability to design better tests and recognize missing test coverage helps developers find bugs. Anything that throws a red bar in our continuous integration system will get fixed.
2. Performing Feature Reviews
We do a Three Amigos–style meeting for every change on my current project. There is usually a developer or two, a test specialist, one or two product managers, and sometimes an ops person, if what we are doing relates back to architecture. The point of this meeting is to resolve as much ambiguity and uncertainty about the impending change as we can. It’s not possible to get features perfectly designed ahead of time, but we want to figure out the important things before the work starts.
Generally, what I find in these meetings is that we have a change that is entirely too big and needs to be broken down into smaller requests so we can think through the problem more effectively. Sometimes there is missing scope in a change request, and sometimes we have a change that seems clear but no one has any clue how we would test it (which means the change isn’t clear at all). Being able to discover these problems before we begin working on a change can help us make a better solution.
3. Requesting Coverage Reviews
Developers have a particular way of testing software. Most of the questions they ask are centered around the themes of ensuring that they built what they think they built (confirmatory tests), and helping them know when they are done with their development tasks. In my experience, this has a tendency to create software with very good first-time quality, but it isn’t conducive to finding more sophisticated problems.
One thing I like to do before I start exploration is go over the current test coverage. I ask my developers to walk me through the tests they built. Generally this will help me ask “what if” types of questions — for example, what if this data attribute we are expecting to be sent from a third-party system is null or isn’t sent to us at all? These questions result in better test coverage at the code level.
4. Building Automation
One aspect of my job is building UI tests in WebDriver for feature changes that need them, then having them run and be checked in as part of the definition of done for a feature change. This bit of process hasn’t existed as long as our product has existed, so of course there are parts that are lacking this type of coverage.
Every once in a while, I have a lull in my work — maybe the development group is working on research projects, or we are on the last day of a sprint and we managed to complete everything in our Jira queue. When that happens, I do a review of our test coverage and build out missing tests.
Last time, I built a test that follows a user workflow and inspects a special part of a page for some data attributes. I discovered while building that test that the page wasn’t displaying data as it was supposed to, and now we had a failing test to remind us of that.
Join 34,000 subscribers and receive carefully researched and popular article on software testing and QA. Top resources on becoming a better tester, learning new tools and building a team.
5. Discovering Testing Problems Early
Most Jira boards I look at give the impression that companies are feature factories. We have a giant conveyor belt and manufacture features one at a time, with all the gory details hidden away. One thing I don’t see there is testing tasks.
The product I am currently working on has a web component and two mobile products. The mobile products were developed well after the web product was made and are consuming the same API to access data. Any change done on the web API has the potential to break or affect the mobile products.
We don’t always know when this will happen because the teams are working independently; this is a vicious dependency tree. We are also missing tests that look at the integration of the web API and our mobile products. When something changes on the web, we have to consult with the mobile teams, get the latest version of each app, and test for any issues we might have introduced in their products. The more software we develop, the longer it will take to test each change. This is a big testing problem.
Several testers noticed this and began lobbying for automation that will provide a faster feedback loop than our current growing cross-platform testing. We are starting to solve this problem now, rather than in a year’s time after we have to hire more testers to cope with the combinatorial problem.
6. Teaching Testing
One of the big tenets of agile and pair programming is that you can create a team of generalists quickly. Each person on the team should, in theory, be able to complete any task. This has a tendency to make a lot of programmers — and a lot of people who approach problems in the exact same way. When I work with these teams, I still tend to see some work that is thought of as “developer work” and some that is “tester work.” In the same way that teaching testers some technical skills can help them be better testers, teaching some fundamental testing skills can help make better developers.
The phrase exploratory testing is shorthand for a very broad skill set that includes understanding heuristics, skilled observation, modeling problem spaces, experiment design and rhetoric. Teaching some of these skill to developers helps them to do effective exploratory testing, and it also helps to get programmatic test coverage with more depth than answering the question “Am I done?”
7. Managing Code Reviews and Pull Requests
Most companies I work with have some sort of process for code review or pull requests (if you’re using Git). Generally, what I see is developers making recommendations so that code is cleaner, is more maintainable or better fits the company code standards.
I like to review pull requests for test coverage. Sometimes, I will notice tests that could be more effective at a different layer in the product. For example, a UI test that is focused on data manipulation could probably be built against the API instead. Other times, I find missing coverage — maybe we aren’t testing what happens when a variable value isn’t set, or maybe there is a missing workflow that seems important. This doesn’t always result in finding problems, but it does make future code changes and refactors less risky.
Task vs. Role
The task of testing doesn’t change a thing about software. No matter how many bugs we find, the software is the software. The role of the tester is more broad than that, though: not everything we do is testing. There are feature reviews, teaching and coaching, writing code, and many other things we do day to day. So, while testing itself doesn’t change quality, testers still have an effect on quality.
What do you do that affects software quality?
This is a guest posting by Justin Rohrman. Justin has been a professional software tester in various capacities since 2005. In his current role, Justin is a consulting software tester and writer working with Excelon Development. Outside of work, he is currently serving on the Association For Software Testing Board of Directors as President helping to facilitate and develop various projects.
Test Automation – Anywhere, Anytime
