This is a guest post by Rachel Kibler.
First things first: as your team adopts DevOps, testing requires a lot more communication and collaboration. People skills are important, and asking questions is important.
There are some other practices that are good ideas for testing in DevOps too, and if your company wants to be successful with DevOps, you should try to implement most of them. Use this quick checklist to identify which devops testing practices you’ve already implemented and get ideas for other ways you can improve.
1. Set the framework
a. Determine your definition of a bug and policy for bugs. Are bugs everything found in production that deviates from expectations? Will they be fixed first or prioritized? How will bugs that require immediate engagement be decided?
b. Make sure your rollback process is easy. (That’s it; that’s the tweet.)
2. Get the whole team engaged in writing stories
a. Split the stories into the smallest releasable (and testable) chunks. API calls that aren’t hooked up to the front end are fine and can be tested on their own before the UI is done.
b. Review the acceptance criteria carefully. Make sure the problem being solved is actually understood. Look for, as Elisabeth Hendrickson says, absolute statements, such as software “always” or “never” doing something. These are potential landmines. Make sure everyone is on the same page and understands the approach.
c. Ask questions about risk — risk to the user, security, business, whatever you can think of.
d. Figure out what needs to be answered with the code change. Write some charters or other kinds of testing notes. Ask for feedback on this. Giving the developers and product owner an idea of how the change will be tested can lead to better code. By “better,” I mean cleaner, more testable code, and a smoother process for the people who need to use it and maintain it.
3. While the code is being written, there are a few things you can do
a. Pair with developers. If you can get a greater understanding of the code and what the changes touch, you’ll do better testing in the end. You can also raise issues during pairing, giving your developers a better idea of what you’ll be looking at.
b. Set up your test environment, or make sure you know how to access the test environment once it is available. Get some scenarios set up, prepare any data you might need for testing, and get users in the right state.
c. Start writing your automated testing, if applicable.
4. When the code is ready for you, it’s time to do what everyone thinks we always do
a. Review the code changes. Make sure the developer wrote whatever automated tests they’re expected to write, whether unit, integration or UI. If there are no tests, ask why.
b. Test the scenarios you’ve set up using all the notes you’ve collected and all the conversations you’ve had.
c. Make sure you touch all the areas of risk. Stuff can change during development, so talk with your developer to see what areas of code the change touches.
d. Run all the applicable automated tests. Typically, start by running high priority or smoke tests. Use automated tests to identify areas you might need to dig in more manually.
e. (If you don’t already know this) Ask your development team if they prefer for you to report bugs you find (one at a time? all at once? in Jira issues? with a summary email?). Depending on how they feel about context switching, they may want you to do as much testing as possible first, then tell them about the issues, or they may want to follow your progress and fix stuff as you find it.
f. Add your automated tests, either in a separate branch or the same one. Make sure you’re following the same code process that your developers are held to.
5. When the code is released, you’re not done
a. Check production to make sure the change got there successfully. Pipelines can be weird, and sometimes stuff gets stuck or doesn’t release properly. Depending on the stability of your pipeline, you might not see this much, but it’s worth checking. You don’t need to reproduce all your testing again; just a simple check will do it.
b. Make sure you’re reviewing error logs to see if there are spikes in errors caused by the change. The amount of time you look at the logs depends on the possibility of intermittency of errors. If you released a change to login, you probably don’t need hours of review. If you released a change to a flow that users see only a little bit, set a query, and come back to it later.
There it is! DevOps is fun and worth it, and with these proven practices, you can set yourselves up for success with testing in DevOps. Check out more of our DevOps blog posts here!
