This is a guest post by Dee Ann Pizzica.
There are issues being reported in the company’s big new software system. This highly technical project has the team’s top brains working on it in a DevOps team populated with accomplished developers. There’s a heavy reliance on infrastructure to keep everything running, and as with any software project, there are also bugs, service disruptions, and outages. The management team wants higher quality, and thus taps the QA team to get involved.
There are many challenges a tester faces in this situation. We’ll start by unpacking how to get started in a project like this, and then we’ll discuss some recommendations you might make for a DevOps team in high-pressure situations.
Start learning
Before you can come in with any recommendations or add value to the team, you have to learn more about the project and all the people involved.
Get a high-level overview
Schedule some time with a team lead or a product manager as soon as you can. In this discussion, you want to leave armed with an understanding of the scope of what you need to learn in order to be successful. Ideally, you’ll get a basic summary of the project as well as each of the team members and their roles.
There are a number of questions you’ll want answers to before you start working with the team. Find out the goals of the project, including any specific upcoming milestones. You should also find out what specific concerns the team has. Are there outages, latency, or incorrect data? They may be incredibly concerned about performance. Get as many examples as they are willing to share about issues the team has faced.
Also, ask them about their expectations for your involvement with the team. You’ll also want to find out if there are parts of the system or team they’d like you to focus on or avoid entirely.
Read documentation
When you’re first getting started, it’s helpful to read through whatever documentation is available. Even if you don’t understand all of the details, you’ll learn about the language the team uses within the project. As you do your homework, you may come across details to give shape to the anecdotes from your overview discussion.
You may also be confronted with a number of concepts that are unfamiliar. Look up terms and tools that you haven’t heard of before and see what you can find out. Make notes about facts that seem particularly important but remain mysterious and confusing despite your best efforts. You’ll need all your notes and questions later.
Join team discussions and meetings
Like when reading the documentation, you may not be able to follow everything when you first join the team’s meetings. Try to take in as much information as you can without being too disruptive to the process. If you start by listening and taking notes, you’re going to learn a lot about how the team has been working so far.
Any time you’re the new person on a project, you have to feel out how many questions to ask out of the gate. If you’re jumping into a working meeting, you should be sure you know the goals and the agenda so that you don’t derail the team. Try to stick to clarifying questions instead of challenging questions as you’re learning and building trust.
Interview developers
Set up a one-on-one conversation with each of the team members, even if you already know them. You may already have an idea of each person’s role because you asked in your overview meeting, but it’s not uncommon on teams to find disconnects between how a person sees their role and how someone else sees it. Don’t miss the opportunity to ask questions about what an individual on the team does on a day-to-day basis and what their part in the project is.
These conversations are a good time to ask some of the questions you’ve been noting as you learn. Ask team members to share their screen and show you how systems work and what tools they use to find information. It helps to stop and repeat back complicated concepts. Not only will repeating help you gain understanding, but it will also reinforce your interest in learning about and participating in the work.
You should steer the discussion to learn what risks and problems your coworkers are occupied with on the project. More than likely they have some concerns and ideas for issues you can explore. This is also your big moment to ask each developer how they test their work currently. Learning the developer’s view on how and what to test is going to give you a lot of information to build recommendations with. You could also ask each person on the team how they think a tester can help them and add the most value to the team.
Make recommendations
Once you’ve spent time learning the goals of the project, asking about the expectations for your role, and talking with each of the team members to see how everything is working currently, you’re ready to get to work. Since you’ve been called in to insert “quality” into a project, there are going to be some errors the team needs to deal with.
Implement monitoring
The team needs to know there’s a problem before the customers do. One of the biggest steps toward stability is understanding where and why there are instabilities in your systems. Monitoring can help with this.
There are a number of tools for DevOps teams. Check out Runscope to monitor API performance. You can start with uptime monitoring so that team members are alerted if certain APIs go down. You can even set up checks for data integrity. This is a great opportunity for you to help the team think of negative tests to recognize bad data that could be indicative of a failure.
Another tool to look at is Grafana. Grafana has monitoring tools that can keep tabs on your databases. You can set up alerts based on CPU usage as well as the response time for APIs. Use your testing superpowers to help developers decide on thresholds for alerts based on severity, as well as how to test a complex set of steps the system may need to execute.
With monitoring, just like with testing, you want to find the most important information first. You also have to weigh the importance of every bug you find, because not all bugs warrant the same level of attention. Apply this thinking to the way the team structures alerts. You need to have a consensus around what alert is worth dragging someone out of bed in the middle of the night, what can wait 24 hours, and what is just excess noise. If you over-notify, you’ll burn out the team and reduce the effectiveness of the tools. Choose your tests wisely so that they are stable, they are valuable, and they alert you at the right time.
Use checklists
One of the tester’s greatest tools is a checklist. The checklist serves as a reminder for lists of information you don’t want to forget, either because it’s highly complex or it’s highly important. If you’re looking to enhance your love of the checklist, check out Atul Gawande’s book “The Checklist Manifesto.”
When you’re making recommendations to a DevOps team, bring your checklists with you. Your DevOps team might prefer to call this collection of checklists a runbook. A great runbook has information about all the procedures that the team needs in order to keep operations running smoothly.
Teams often neglect the runbook because they’re busy dealing with emergencies, so they don’t take the time to write down the steps to fix an error, instead of storing that information in their heads. This creates risk in many ways — a team member could leave the organization and take everything they know right out the door with them, a team member could go on vacation or take leave, or a team member might be burnt out or stressed and miss an obvious but important step in a process because the stakes and the pressure are high.
If you find the team has a runbook, that’s great. Do some digging and ensure that it’s accurate, up to date and thorough. If not, help them get one started. While it takes time to document processes, the right checklist just might save you from disaster.
Automate
Reduce the risk of human error and increase your reach by automating as many tests and processes as you can. Even if you aren’t an automation tester, you can still bring value here. Even if you don’t know how to write the code for the tests, you do know how to think about what makes an effective test.
Work with the development team to identify a set of tests that will aid in each phase of the project. Since your DevOps team likely has their eyes set on continuous delivery, you want to have a test suite that covers all of the core functionality that runs whenever changes are merged. You can also explore specialty tests that are just for certain features that may give you a lot of value out of the gate but are so specific that they aren’t worth the effort of maintaining. Curate your automation suite frequently, and consider additions based on important issues raised by customer concerns or outages.
Go forth and be awesome
Remember the most important message in this scenario: You belong here, and you can make a difference on this project. It’s easy to be intimidated by a steep learning curve, but give yourself a pep talk and dive in.
As a tester, you have skills that have prepared you: You listen, you explore, you learn, and you question. You bring skills as a collaborator and a thinker, and you know how to break software. Use these skills to your advantage, and speak to your team to learn how you can best deliver value. Learn more about DevOps on our blog.
