This is a guest post by Carl Shaulis
Perhaps your product is new; perhaps the accelerated world of DevOps led you to believe testing planning was not required. Perhaps the old toolset you were using, the spreadsheets and whiteboards, simply don’t scale as the team does. For whatever reason, you wake up one morning and realize that you need a better tool to manage the test activities for your company. As no one else will do it, you volunteer to pick the tool.
Now what?
Exactly how to go about this task can be shrouded in mystery. Even if you have had a long and sustainable career testing software, the last test planning tool replacement was probably done by someone else. The decisions and processes are undocumented. Even if you pick a tool that works for one test approach, it might not work for all. For example, the human test script results might be stored in a spreadsheet, while the automation results from the user interface may be stored in an Elastic Search cluster and the unit test results in an AWS Simple storage service, S3. This gives no single, unified view of the status of the software. The next-generation Test Management Tools promise to bind these together — but will they work for you?
Today, we offer you a quick checklist to get started. You might not need all of these, and you might have some hidden requirements. This checklist should be good enough, enough of the time, to narrow the selection down and figure out what else to consider. Use this test management checklist to score, rank, and limit test tools.
The primary objective of this checklist is to narrow down the selection process from dozens of attributes to a handful so the team can demo, trial, and consider the best tool.
Now let’s get down to business.
- Structure
The test management tool should align and be flexible enough to handle the needs of the organization. A multi-division company will have a structure. For example, each business unit may have products. Each product may have test suites. The test suites consist of test cases. Test cases are executed in runs, and so on. This implies a hierarchy. At the high end, a tool may be ultra-configurable and fit for a Global 50 Corporation. At the low end, a tool might have the ability to build suites for a single product. The question here is does the tool map into the scope for your company. For example, you may want to structure your test plan based on the layers of testing (e.g. – Acceptance, Integration, User Interface). You may want to separate test runs for automation from manual or have a place to record human exploration in sessions. Test plans might break down into suites or cases, which could divide into features, functions, or correspond to user journeys. The structure within the tool should be flexible enough to meet the needs of your teams and the business domain the solution is supporting — along with a little room for change and growth. Again, the question is not “is this amazing” but instead watch a quick demo and ask if the structure will fit the team.
- Access and Ease of Use
Most test case management tools today use a Software As a Service (SaaS) model. That means no software to install, but someone has to set up users and configure permissions, removing users when they leave the company. Ideally, the tool fits into a company’s single sign-on solution and the licensing permits access for everyone. During your validation of the best tool for your company, you will want to consider licensing, because it could have a direct impact on accessibility and cost.
Everyone in the company should be able to find, view, and contribute to a test plan or test cases. The User Interface should be intuitive to the community. At a minimum, the engineering and testers should be able to rapidly generate a test strategy as a digital artifact.
3. Software Engineering Process Integration
Modern organizations store test data in disparate places. There is setup data, test code, the overview of what tests to execute (the test cases), the charter and notes from exploration, and the results of the test runs. Those test run results could include defect numbers or references back to the code.
The gives us at least five potential integration points:
- The requirements that each test exercises (Jira, VersionOne, Trello, Rally, LeanKit, and so on)
- The defects found by the test run for that test (Bugzilla, Zoho, Mantis, FogBugz, Jira)
- The product code that was exercised (Version control)
- The test code that was exercised for this test case (Ranorex, test automation code, Version Control)
- Any “recipes”, “howtos”, or instructions to supplement the test information (Wikis, Confluence, Sharepoint)
Not all companies need all these things to connect. The question is how much traceability from one to the other should the tool have — and does it have that traceability?
This is a fitness for needs question. Before adopting a test case management tool, determine what the engineering workflow will be including all integration points. The users will consider what information they need from the Test Management solution and what key signals get pushed to the Test Management solution. The team must make sure the workflow is possible and data flow as expected.
Finally, if the Test Management system does not have great reporting and insight capability (#8 below) but it can export data into another system that does, that may be good enough.
4. Continuous Integration and APIs.
While this is similar to the previous item, we broke Continuous Integration into its own bullet as it is so crucial. Companies that pursue a CI approach will want the CI system to be able to update the test management tool and keep the two in sync. One way to do this is with a robust API, where other computer programs can do the same thing that the users can. Some tools may have a command-line interface as an alternative.
This feature can be a strategic driver. The ability to import automated results mapped to the test design elevates the team, which enables management (see Insights and Reporting) to visualize all testing activities in one place.
Be sure to know if you need CI integration or APIs and if the tool can meet your needs.
5. Scalable
This category is subtly different from structure. Structure is concerned with mapping the constructs of the tool onto the business domain. Scale considers if the software can actually handle the sheer volume of test result data. If a typical product has a thousand test cases and the company has a hundred products and is doing continuous deployment, creating test runs for every case for every version control change … that is a lot of test run records.
A simple list of customers of the tool and a few minutes of analysis should tell you if the tool can be a fit for your business.
6. Insights and Reporting
If you are a test manager, you are constantly being asked. “How is testing going?” “Are we ready to release?” “How many critical defects do we have?” “Upon analysis do we have any constraints or high-risk services?”
Every Test Management Tool must generate reports. The question for this checklist is: Are those reports good enough to get the reader a real grasp of the situation? Consider the possibility of exporting or publishing the results as a web page. Instead of asking the test manager, any stakeholder from senior management to the receptionist can simply press refresh in their browser!
Now consider if the tool can predict completion based on data, show visualizations of coverage, or visualizations of quality. This might change the conversation from “why is QA taking so long?” to “why are there so many bugs for QA to find? Why is there so much retesting?” This is an entirely different, healthier question, that indicates management understands the status of the software under test, not just the schedule.
Dashboards should also be configurable at any level of your organization. The CTO may have a different expectation than a director. Each team will have their own view into their ecosystem, or the ability to zoom down – or up – to the right view.
Another powerful tool is trend analysis with configurable time durations. Looking at defects (or pass rate percentage) for a week may tell a different story than reviewing an entire quarter. Executives like to see information, trends, and insight in quarters or even as a month-over-month comparison. You will not be pleased if you have to export all of the data and then generate pivot tables and graphs in a different tool.
Any Test Management solution should provide the insight for teams to quickly improve their process or pivot their work.
7. Aligns to DevOps
The modern DevOps pipeline does a lot more than just having humans explore the user interface at the end. For a unified view, the Test Management tool might need to provide insight into Unit tests, integration tests, performance tests, security tests, accessibility tests, and end-to-end automated tests. If that data includes timestamps, and insights into performance changes over time, it can be valuable to Site Reliability Engineers and other DevOps roles.
Breakdowns of what types of tests are growing more quickly can help the team adjust their effort to a “sweet spot.”.
8. Simple Migration Paths
How does your team go from whatever it is doing today to the new tool? If you can create spreadsheets and import them to at least create the shell of the test plans, suites, and cases, that will make life a lot easier. Any tool you choose should enable the import and export of test cases in standard formats such as .csv, .xls, and .xlsx.
For that matter, if you want to leave, the tool should at least provide an export path or create read-only PDF documentation. The ability to query the database in SQL might be a plus — if you need it. Remember, the checklist is fit-for-use for your organization, not a list of bells and whistles.
9. Manage Test Cases
Products and teams change. This is a fact of fast-paced dynamic companies. You will need the ability to archive test cases or even restore them. You will need the flexibility to move test cases from one team to another team. You will need to adjust the priority and scope of each testing cycle to align with the software being delivered. You might want the ability to create separate views by team and then change the team people belong to, or create a variety of reports.
Occasionally teams are forced into compressed timelines so full segments of testing may be paused or skipped based on the risk to your customers.
10. Cost
Today the cost of a great solution can make or break a business decision. Your company will need to determine if productivity gains and data accuracy warrant the cost. Since there are many Test Management solutions in the comparison table, the cost can be the differentiator if all of the other attributes are comparable.
Cost should not be the most critical attribute in the matrix, but if scores are comparable and the difference in cost will break the budget, then the choice is clear. Our suggestion here is to calculate the cost of a year and consider that when it comes to a purchase decision.
Run the checklist
As you prepare to find the right Test Management Solution for you, download this test management checklist, and fill it out. For each category, fill out a simple score:
10 Ideal for our organization
5 Probably would work
0 Feature missing
-10 Important feature missing
-20 Critical feature missing
-90 We should not consider this product due to missing feature
These scores are weighted, intentionally, to be most likely to align with your success. For example, if the scale or structure will make the product a no-go, it should be scored that way. Without that kind of scoring, it is possible that the product is all 10’s in other categories and “wins.”
Of course, this is a guideline to limit your selection of Test Case Management Tools. The next step is the tough work of trials, demos, and putting the tool through its paces. The proof of concept is the key to success and may require collaboration and close support from the solution provider.
Selecting a good test case management tool is difficult, but it shouldn’t be impossible. We hope this report made it a little easier. And don’t forget to download this test management evaluation checklist!
