This is a guest posting by Nishi Grover Garg.
Agile teams constantly face an overload of regression testing. As sprints progress and new features get built into the software, agile testers are faced with the challenge of testing all these new features while ensuring nothing from the previous sprints breaks as a result of the code changes and defect fixes. This piling regression can easily become a nightmare if not planned for properly in the beginning.
Planning for Regression
Every team will have a different approach to regression. The intensity of regression tests needed will be decided based on factors like the complexity of the system, the number of interfaces with other systems, the size of the change being made, and the safety-critical nature of the software. The agile team’s regression strategy must be decided at the onset of the project, and effort and time allocation must be done in the release test plan. Further, every sprint test plan must have an allowance for a regression effort in addition to the sprint-level testing to be performed.
Regression strategies vary from exhaustive to selective, based on the nature of the project and the context of the current release cycle.
Teams may decide on a “retest all” strategy, where you rerun all test cases for the software or feature at hand as a part of regression.
Test selection is where you select some of the test cases for regression from the entire pool of test cases based on the functionality covered or modified. This decision may be made by the test manager or the testers, depending on their knowledge and experience.
Prioritized regression is where you select the test cases for regression based on their priority in the context of the cycle. For example, for a major release cycle, you may decide to rerun all high- and medium-priority test cases for the entire application. On the other hand, for a hotfix or a patch release, you may decide to run only high-priority test cases for the modified feature during regression.
Deciding the regression strategy for your agile team is the most important step in the initial phases of the project.
Performing Regression Using TestRail
TestRail is a test case management tool. It makes regression testing easy by providing you with a repository for all of your tests, ways to track changes to those tests, results associated with the tests, and ways in which to plan, sequence, and execute all of your tests during the various phases of your approach, including automation integration if this is a part of your strategy.
By default TestRail comes with a set of fields that are useful for structuring, organizing, and compartmentalizing test cases. But you can customize the fields to record any additional information that your team needs.
Test case history is available to see how many times tests have been executed, what the results were each time, and any additional information, like attachments or defects, found.
All recent test results for a test case also can be found at a glance in the test results tab.
When starting test runs for a test suite, you can decide to include all or selected tests for that cycle. So whenever you need to test a new version of your software, you can simply start a new test run against your existing test cases and include a number of tests from that test suite based on your regression strategy.
Selection of Regression Test Cases
TestRail allows testers to manually select test cases or entire test sections based on the changed areas or functionalities that will get included in the test run.
It also allows testers to select the test cases for a test run based on multiple filters, like priority of test case, type, title, or area. For example, if you do not have enough time for revisiting all test cases of a particular area, you can select test cases of the highest priority only. Or let’s say you would like to run only the test cases previously marked as regression tests — you can select the filter for Type as “regression.” And a combination of both of these filters would select all regression tests with a high priority for this test run.
This allows testers to reuse the test cases already present for their regression cycles. There is no need for creating a separate regression test suite or duplicating any tests for these types of runs.
Performing Regression on Different Platforms
Configurations and operating systems can be added in TestRail for repeating the test runs on different platforms. You can even start multiple test runs for different platforms, including operating systems or web browsers you need to test against.
TestRail automatically replicates the test run for the selected number of configurations and provides a comparison of run results on different platforms once executed.
Monitoring and Reporting your Regression Test Cycle
TestRail comes with built-in reporting options, and many of the commonly needed metrics for agile teams are available within the interface. It also integrates easily with defect tracking tools like Jira, Redmine, and Bugzilla, so any defects found during testing can be specified with the failed test case itself.
All test changes, results, comments, and attached files are displayed in a single place, maintaining the history of the test case.
Monitoring the progress of your test run cycle is made easy by the visual representation on the test run page, which shows the number and percentage of tests executed, along with pending tests, with their results status. Testers can easily mark the results of the tests executed directly from this page, too.
A test manager can create print and PDF reports for the test cycle in progress in one click, saving time and effort for the collection and collation of the information in order to share it with stakeholders. They can also export the report as various spreadsheet file types to further analyze the data.
TestRail also has a forecasting feature that uses your historical time data and entered estimates to predict how much time your test will take to execute and how much of the remaining effort is needed for your test run. This is represented in the familiar burn-down chart format on the progress page, along with a projected completion date.
Extensive reporting options are also available, along with ready templates to generate summary reports, comparison reports for multiple test runs, coverage reports and defect charts.
With all these features, TestRail can ensure smoother planning and execution of your regression testing, with complete transparency of progress and test results of each test case. Tools like TestRail can take the overhead of management away while keeping your team completely focused on their quality goals.
To learn more about TestRail, you can watch this video series, or spin up a trial instance to play around with the tool and get a better feel for TestRail’s capabilities.
