This is a guest post by Raj Subrameyer
Organizations are striving to release high-quality products to customers faster than ever before. There is considerable pressure to stay competitive in the current market. A popular agile approach that enables this to happen is DevOps.
DevOps helps to make the development process leaner and a key component that determines its successful implementation is Continuous Testing (CT).
What is continuous testing?
Historically, QA has started testing the application only when features were pushed to the QA environments. The problem in this approach was that defects were found late in the development process, at which time they proved too time-consuming and expensive to organizations.
With continuous testing (CT), testing is performed early, right from the planning phase. It continues throughout the development process until the production and monitoring phase. This aligns with the “shift-left” paradigm.
With CT, teams get immediate feedback on the application at each stage of the development process. Automated tests run each time a code gets checked in, so teams get quick insights to make informed decisions.
CT also reduces the risk of finding critical defects in production that could cause unnecessary cost, time, and effort for the organization. The key is to test as early as possible and find defects as soon as possible.
Get Your Continuous Testing on Track
1. Identify gaps in the existing test process
Before planning steps to implement CT, you need to get a handle on the existing problems that CT could help solve. Look through your current testing artifacts; your test process, which may include manual, automated, and exploratory testing; the types of production defects found; and the bottlenecks that are preventing the timely release of features with high quality.
This information will help determine how your team can implement CT and what tests need to be included and run in the different stages of the development process.
2. Set up the infrastructure
Implementing a seamless CT pipeline involves having the right infrastructure to support the effort. This may include:
- Realigning the teams
- Unifying development, QA and staging environments with production-like data
- Changing the deployment process
- Setting up automated tests to run each time code gets checked into different branches
- Using service virtualization to test APIs
- Having CI/CD tools such as Travis CI to automate build, deploy and release cycles
- Implementing containerization using Docker and Kubernetes
- Having a cloud infrastructure to deploy and run tests quickly
- Setting up monitoring tools to get more insights into the development and release process
3. Identify the right tools and resources
Teams need to invest in using the right tools for various activities. This could include:
- Testing tools and frameworks for unit testing, UI testing, API testing, performance testing, security testing and accessibility testing
- Version control tools
- CI/CD tools
- Monitoring tools
- Security scanners
Teams also should have skilled employees who can use these tools effectively throughout the development process. They should be capable of analyzing reports, troubleshooting problems, and fixing issues as soon as they occur.
4. Give sufficient training
Having an effective CT pipeline involves a mindset shift. Instead of thinking of testing as something that happens only during a certain time, you need to reimagine testing being an integral part of every single stage of the development process.
This means sufficient training should be given to people to help them adapt to the new way of thinking. Teams have to embrace that shift-left testing will be the norm moving forward.
5. Automate as much as possible
Automation is the glue that binds CT and the DevOps pipeline. When set up the right way, automated tests help reduce the time between release cycles and make them more dependable.
Everything that can be automated should be automated. This may include unit, deployment, integration, smoke, regression, security, and performance tests. Testers play an integral part in the DevOps and CT process by starting automation early in the process so defects can be found sooner rather than later.
6. Continuously monitor results
It is wasteful to set up everything for a seamless CT pipeline and not measure the progress to know whether the implementation is successful.
Here are some steps you can take to continuously monitor the effectiveness of your CT implementation:
- Use daily standups, team meetings, retrospective meetings, and other channels to gather feedback on what is going well or what can be improved
- Have weekly check-ins with your team to analyze and discuss the results of the CT process
- Come up with key performance indicators (KPIs) to measure progress
- Make changes to the CT pipeline based on the gathered feedback
Setting up a good CT process is an iterative approach, and the more you look at the data, the better insights you will get into optimizing the process.
KPIs to track for product quality
Since CT is part of the DevOps process, the metrics that apply to the DevOps process apply to CT as well:
- Test case coverage
- Unit test coverage
- Pass/fail rate
- Number of tests executed
- API pass/fail rate
- Requirements covered by tests
- Blocked test cases
- Percentage of automated test cases
- Successful code builds
- New defects
- Critical defects
- Automated tests prioritized by risks
- Code coverage
- Release deadlines
- Total number of defects discovered
- New API defects found
Using these metrics, teams can measure gaps in the implementation process and take corrective actions immediately.
Tools needed to implement continuous testing
There are various tools and frameworks that enable continuous testing in different phases of the development process, including CI/CD tools like Travis CI and Jenkins, automated testing frameworks like Selenium or Appium, automated testing tools like Ranorex, unit testing tools like JUnit, NUnit, and Jasmine, monitoring tools like Splunk and Graphite, and test management tools like TestRail.
In particular, test management tools help with viewing details of all the testing performed in the development process on one single dashboard. These tools can then generate reports to be shared with other stakeholders. It is critical to choose the right test management tool that will facilitate collaboration within teams and track the overall testing progress.
For example, when you use TestRail in a CT pipeline, the report generated may look like this:
Image: A sample report generated after automated regression tests are triggered by code check-in
Continuous Testing plays a vital role in the successful implementation of a DevOps pipeline. Organizations must identify gaps in the current testing process, use CT to fill these gaps, and then transition into DevOps. This helps to catch defects as early as possible and deliver high-quality products at a faster rate.
