There are some approaches and practices that will help your team get the most value from TestRail. In a recent webinar, TestRail product manager Simon Knight shared his “pro tips” to help test managers/test leads optimize testing workflows for any development approach: waterfall, agile/scrum, or DevOps/continuous.
The key points from the webinar are summarized in this article. Use the links below to jump to the topic that most interests you. You can also watch the full webinar on-demand.
- Requirements Traceability 2:30
- Test Case Templates 8:00
- Test Cases 15:33
- Test Runs 17:45
- Test Plans and Milestones 30:45
- Test Management 34:00
- Q&A: 40:45
Test management is “the set of testing activities necessary to support the successful release of a product, including test planning and design, test execution, test function administration (such as test maintenance), and reporting. TestRail is designed to support all of these activities.
Requirements Traceability
As a test case management system, TestRail does not directly support requirements for the purposes of linking them to your test cases. But, it does integrate with a significant number of requirements management tools, such as Jira, Assembla, or GitHub (yes, even if you’re using Word or Excel to manage requirements). Use the TestRail test cases “references” field as a direct link to your requirement. TestRail test runs can also have references. TestRail has dedicated plug-ins for Jira and Assembla for extended requirements support.
Test Case Templates
TestRail offers several templates for setting up your test cases. You can also set up your own custom templates if none of the default templates meet your needs.
If you’re following a sequential, waterfall-style development approach, then you might want to use the Test Case (Steps) template shown above to set up your test cases. The Test Case (Steps) template offers flexible test script definition, and allows you to decompose scripts into steps and expected results. Each step gets its own status at runtime. In addition, this template is ideal for business or user acceptance testing (BAT/UAT).
If you’re using more of a Scrum or agile approach, then the Exploratory Session template may be a better fit. This template offers a loose framework for test definition and aligns with various exploratory approaches. The approach that you want to follow with exploratory testing is to identify a set of exploratory testing coverage, execute that initial set of session-based tests, and then iterate as you learn more about the product. For additional information, there are several resources you can refer to, such as Exploratory Software Testing by James A. Whittaker, Session-Based Test Management by James & Jon Bach, and Thread-Based Test Management by James Lyndsay.
For a DevOps/Continuous approach, the preferred test case template is Test Case (Text). That’s because your release artifact is generally considerably smaller than in the other approaches, perhaps consisting of only a single pull request. The Test Case (Text) template works well as a placeholder for automated test activities. You can use the TestRail API to pull/push information from and to TestRail.
Test Cases
The general principles of managing test cases in TestRail are as follows:
- Organize by component or functional area
- Prioritize your test cases
- Experiment and then customize your test cases
Test Runs
Once you have your test cases created, you’re ready to start creating and executing test runs. The specifics vary somewhat based on your development methodology.
If your team is following a sequential/waterfall approach to development, you have several options to organize your test runs (using the Test Plans & Milestones capabilities of TestRail):
- Split test runs by functional area/component.
- Split test runs according to the type of testing
- Split test runs according to the priority of testing.
For an agile/Scrum approach, consider organizing your test runs as follows:
- Feature testing
- Exploratory testing
- Automated regression testing
For a DevOps/Continuous approach, use the TestRail API to create runs and report test automation results. You only need two endpoints: the add_run endpoint, passing in the IDs of the test cases that you want to execute. Use the add_results_for_cases endpoint to add the results for the cases in your test run. You can also use those endpoints to pass in results for milestones, and add references to your requirements in those runs.
When setting up your automated testing test runs, it’s helpful to do the following:
- Create a test run for each execution of your automated test suite
- Group them under milestones or plans, for aggregated reporting
- Close your runs once they’re completed.
You can choose to get your automated results in real-time (by refreshing the page) or use a bulk API endpoint to get all of your test results in a single pass, which is an easier and more performant way to use the TestRail API for your automation.
Test Plans and Milestones
From a functional perspective, test plans and milestones to organize your testing efforts are entirely optional. You can use TestRail with only test cases and test runs. But organizing your runs into plans and milestones has benefits for your status and reporting efforts:
- Results from all of your uns roll-up into plan stats
- Results from all of your plans and runs roll-up into milestone stats
- This streamlines your reporting.
TestRail does the report configuration for you, by automatically populating the summary reports based on the TestRail view that you used to navigate to that report.
Test Management
To this point, the webinar has addressed how you can think about test management in terms of workflows depending on the methodology that you’re using: waterfall, agile/Scrum, and continuous. While there are similarities, the feedback loop: you’ve got requirements that link to test cases, the test cases are linked in runs organized as needed for your specific context; and then grouped under a release version or milestone. Generally, less organization in continuous due to the shorter feedback loops.
In summary, here’s the TestRail approach to test management:
- Maximize the value of test cases with templates and traceability
- Organize your tests into runs, plans, and milestones based on development methodology
- Used advanced features in TestRail to plan, execute, and report on your testing.
If you’re mainly following a sequential/waterfall approach, here are some test management tips:
- Use references to provide requirements traceability on your test cases
- Use the Test Case (steps) template for your scripted test cases
- Organize your runs according to testing activities (functional, regression, UAT)
- Organize your runs into plans and milestones to aggregate and report on your test progress and results more easily.
If you’re mainly following an agile/Scrum approach, consider doing the following:
- Use references to link to features, epics or cards
- Exploratory Session templates can be used for exploratory testing approaches
- Leverage dynamic filtering to facilitate iterative testing
- Roll-up test results into plans/milestones for better visibility and reporting
Finally, for a DevOps/Continuous approach:
- Use test case references to link to PR’s and tickets
- The Test Case (Text) template can be used as a placeholder for automated test activities
- Use the TestRail API to create runs and report automated test execution results
- Roll-up test results into plans/milestones if it makes sense to do so
If you haven’t yet tried TestRail for test case management, start your free trial today!
