This is guest post by Ostap Elyashevskyy
If you are building your QA process from scratch or improving your existing QA process, choosing the right test management tool is crucial to help you find bugs, manage test cases, and ultimately, streamline the software development process.
In this post, I will share the experience that our team at ELEKS had searching for a test management solution that would work for our team, some alternatives we evaluated along the way, and why we ultimately chose TestRail.
Common Test Management Challenges
Some of the most common challenges you may experience at the start of a software development project include:
- Lack of visibility
- Lack of test management tools or low usability on the project
- Incorrect or inefficient use of any current test management tool
- Poor quality test cases that are hard to maintain
- Lack of Jira and test management tool integration
- Lack of traceability between requirements, Jira artifacts, and test cases
Lack of visibility
It is easy for testing work to be invisible. I’ve seen so many occasions where testers do all the work, but the results stay locked up on their minds or on their machines, not helping the project at all. Testing is critical in any software project, but only if the results are communicated effectively and quickly.
QA is not only about running tests. It’s just as important to generate visibility for your team via good reporting and dashboards. Ensuring visibility means understanding the answer to questions like, “Who is testing?”, “What functionality is being tested?”, “What is the scope of testing,” and “When will we finish all testing activities?”.
Lack of test management tools or low usability on the project
Oftentimes, when teams do not have the proper test management tools in place, they experience poor usability throughout their projects. Many projects track their test cases and checklists in spreadsheets. While spreadsheets are an option for smaller projects, they are unsustainable for larger projects with multiple engineers. With spreadsheets, updating and managing test data oftentimes becomes a time-consuming and error-prone nightmare.
Incorrect or inefficient use of any current test management tool
If you choose a test management tool that does not align with your organization and its practices, you are at a high risk of wasting time and money. When a test management tool is not used to its full capabilities, it is most likely the wrong tool for the job. The wrong tool can wind up simply as a means of test case storage or another wasted tool in your company’s collection of unused software.
Poor quality and maintainability of test cases
Teams often encounter two core problems with their test cases: they don’t have enough contextual information, or they are written in a way that makes them hard to reuse.
For example, test cases exist but there is no information about priorities, versions, configuration, references to requirements, or other important information. This makes it difficult to prioritize the right tests, test accurately, and report on the results of testing in a way that’s meaningful to the rest of the team.
Lack of Jira and test management tool integration
Check whether your test management tool integrates with your existing tool stack. In particular, can it integrate with your bug or issue-tracking software, such as Jira? The advantage of such integration is that users can easily link bugs to test case runs and benefit from advanced traceability. Lacking the proper integrations can prevent test cases from being executed in the scope of a feature.
Lack of traceability between requirements, Jira artifacts, and test cases.
The goal of traceability is to help you plan and manage testing activities (including defect management) to minimize risk better. A lack of traceability can create problems throughout the software development lifecycle (SDLC). For example, lack of traceability can make it difficult to know the impact of changes, resulting in inconsistencies that can derail your testing progress, waste time, and deplete resources.
Selecting a test management tool
Having an established test management tool does not mean your problems with traceability, visibility, and documentation are automatically solved. I experienced this firsthand when using spreadsheets as a test management tool and TestLink as our test management tool before switching to TestRail.
Limitations of TestLink
Let me elaborate — although TestLink is a free open-source test management tool, it does not offer feature advantages over other test management systems. TestLink has poor usability, and the Jira integration is one-directional. TestLink’s Jira integration only allows you to:
- Link Jira issues to the test case being executed
- Create issues when a test case fails
Other limitations you may experience with TestLink include:
- Poor usability
- Outdated UI/UX
- Longer onboarding
- No dynamic visual dashboards with the test status and statistics
- Difficulties customizing extra fields and statuses
Zephyr vs. TestRail
If you want to work with the most user-friendly and powerful tool that fits the needs of your team, I recommend you to install trial versions of the tools you are scoping.
Personally, I tried both Zephyr and TestRail in depth and chose TestRail as our test management tool. For better visualization of the pros and cons, I created a table that summarizes my experiences and will hopefully help you to choose the most optimal tool for your team.
Zephyr pros:
- Good traceability between different Jira issues types (stories, bugs, test cases)
- Because it’s an app built into Jira, you can create test cases as Jira issues
- Great reports and customization options
Zephyr cons:
- Requires you to pay for each user in your Jira instance (even if they don’t need Zephyr)
- Performance issues (both on UI and API levels) affecting all users in your Jira instances
- Ability to share steps is not supported in all versions
Ultimately, because of poor usability, lack of visual dashboards, and a pricing model that did not fit our needs, Zephyr was not a good fit for my team.
Why we chose TestRail
We chose TestRail as our test management solution because of the following benefits:
- Two-way integration with Jira
- Improved visibility
- The ability to share the same set of test steps across multiple test cases
- Actionable software testing reports
- TestRail’s usability and intuitive user interface
- Flexible custom fields for test cases and results
- Ability to assign test runs
- Test case templates and test plan features
TestRail and Jira integration
TestRail integrates with Jira in many different ways. You can easily push bug reports to Jira, look up issue details, jump to Jira’s New Issue form and more. Moreover, teams can extend TestRail’s defect plugin for Jira, enabling them to adapt the integration to their Jira customizations if needed.
The Jira integration for TestRail provides better visibility and traceability than other test management tools.
It also allows you to easily answer questions like:
- What feature does not work and why?
- Do we have test cases for this feature?
- Why did these tests fail?
- And, what is the progress of release/milestone testing?
TestRail supports two-way integration with Jira so you can:
- Link test cases and test results with Jira
- Create Jira issues from the test management tool (you may also want to customize the issue template)
- Check the test results and cases directly from Jira issues
- Build traceability reports with information about test cases, test results, and referenced Jira items
- Preview Jira issues from TestRail
How to integrate TestRail with Jira
Here are the two steps to configure the Jira test management integration for TestRail:
- Set up the integration in TestRail
- Add the TestRail Jira add-on to your Jira instance
TestRail integrates with Jira Cloud, Server, and Data Center. Here is TestRail’s Jira integration documentation with setup instructions for each platform:
Let’s check the main features which can help you to improve visibility:
- Add reference(s) to test cases in TestRail
In the picture below, test cases are associated with the Jira subtask, which was created to track test design activities (in my case).
If you open the “QA – Test Design” subtask with id: DESK-10 (in my case) in Jira and go to the “TestRail: Cases” section, you can see the associated test cases with it:
- Add Jira reference(s) to test runs in TestRail.
If you create a subtask or another issue type in Jira for testing purposes to run some tests (Ex: “QA – testing feature”), you can create a test run in TestRail and associate it with a Jira issue (subtask).
Now, if you open this subtask in Jira, the result of the test run will appear inside the Jira issue, so you don’t need to open TestRail for it (but you can open test results in TestRail as well):
- API integration
- Automatically updating test statuses in TestRail during automated test execution can save you time and improve visibility into your test automation.
- If you discover any bugs while reviewing your automated test results in Jira, you can use TestRail to link them to defects in Jira to ensure full traceability—even with automated tests.
- Before enabling and using the TestRail API, ensure that your tests are stable and results reflect the actual state of your application so the results are trustworthy.
At ELEKS we use Python API and our test automation engineer, Maksym, implemented a wrapper over TestRail API which adds extra functionality. So, if you are using a Python client – you can find original Python API bindings or Maksym’s API bindings here.
- Import API bindings and configure them: provide URL, API keys, and other information needed according to API documentation
Other helpful TestRail features
There are many other TestRail features that we found useful for test management.
Shared Steps:
Shared steps allow you to reuse common steps across test cases and improve the maintainability of test cases (not to mention saves you time!).
Reusability is achieved by “importing” shared steps. Be sure to carefully consider your naming convention for shared steps because you are likely to have hundreds of shared steps. Using a pattern for naming shared steps (for example, <functionality or page> – <step description>) will help you to navigate through steps and import them easily.
If any changes are needed, you can update it in a single place and other test cases will reflect the changes.
Dashboards and reports:
TestRail’s dashboards and charts represent the current status of testing. You can share them at sync meetings or send them regularly to give stakeholders a quick overview of testing progress.
The level of detail in the reports can also be configured in the “Reports” section.
Reports are comprehensive and can be generated according to 15 predefined templates sorted by categories: “Cases,” “Defects,” “Results,” “Summary,” and “Users.” You can use templates from the “Summary” section to generate a universal report that can be helpful to everyone from QA engineers to project managers and product owners.
Ability to assign test runs to Quality Engineers and track current assignments:
If you are a QA lead – this feature allows you to distribute testing across the team and track the progress.
Customizations:
You can customize your test case, result status fields, priorities, templates, and other fields. In our case, we added additional result statuses below to provide more information about the test run:
- Environment issue – issue with environment or configuration
- Changed requirements – fails due to changes in requirements
- Known issue – we are aware of this issue
- Next release – should be moved to the next release
- Skipped – other reason to skip the result
Test case templates:
When time is limited, it can be cumbersome to describe all steps in a test case. To save time, you can use a simplified template for test cases or even an exploratory session with a description of the functionality you want to verify.
Additional recommendations to improve the QA process:
- Create a separate Jira board (e.g., Kanban board) for one-off QA tasks outside of actual test case writing, planning, or testing to increase visibility into your progress and workload
- Capture and monitor metrics on Jira dashboards:
- Number of open bugs per release/milestone
- Critical and blocking issues
- Number of bugs that were found during last week/iteration
- Number of bugs found by automation
- Test coverage (see more about this in this Traceability and Test Coverage in TestRail)
TestRail’s two-way Jira integration provides you with the ability to see the execution status from TestRail and Jira and allows you to preview a Jira issue when referenced with TestRail result or test case.
TestRail’s usability and intuitive user interface allow for quick onboarding and the flexible licensing model allows you to pay as you go. This approach worked well for us as we started with 2 users and then scaled up to over 20 active users in the system, including developers who use TestRail daily as their source of truth for test cases, data, results, and overall progress.
TestRail allowed us to streamline our test processes, leverage integrations with Jira, and improve visibility and usability. Ultimately, by using TestRail as our dedicated test management solution, our QA team was able to gain visibility into testing, scale QA more efficiently, and accomplish more testing.
