Guest post by Emanuele Ciurleo
As a tester, you design tests to validate the requirements for your application as understood at the time, and often, a particular version of the software at its point in the development cycle. The tests will take into account functional and non-functional requirements, but also try to mitigate potential risks in the system being tested. If you know the development of a particular part of the system involved a lot of complexity, you may write an extra test or two based on a scenario that was never a documented requirement. This way, your test suite provides coverage as robust as your team’s understanding of the potential risks. This applies to tests written as part of test-driven development or automated tests that form part of a Selenium suite. When you wrote that test, it tested what needed testing. Perfect!
But software constantly evolves, especially in an agile world. Your understanding changes, requirements change, and invariably, code changes!
This can often mean the well-crafted suite of tests you (and your team) wrote is no longer appropriate for new versions of your application. If a dropdown changes to a radio button, you need a new set of tests for it, and the old ones can be removed. So back you go to the tests to update them to match the new needs. Nice and easy.
Imagine, for example, changing a test of minimum character length on a “Surname” field from 3 to 2 from version 1.4.2 to 1.4.3. The test passes against the old version, but why was it changed? Does that change have unexpected consequences? Is it appropriate to run this test against version 1.4.2? If you’ve written a suite of tests for 1.4.2 but those tests are changed for version 1.4.3 of the software, how do you test a rollback scenario in the event of a deployment failure? You need to keep the tests from version 1.4.2 as well as 1.4.3.
Understanding how and when a test case was changed helps you understand the context and the coverage expected by the test case. The tester who wrote the test might be clear on why the function was tested in the way it was, but if a new team member joined today, would they be able to understand? Having a version history will allow the new team member to understand exactly what the test was originally designed to cover and why and how it has evolved over time. It also helps the testers ensure that multiple small changes over time haven’t caused the test to no longer cover what it was originally written for.
If you work in a highly regulated environment, it can be especially important to keep previous versions for mission-critical assets to meet regulatory compliance standards. For example, ISO/IEC 27001:2006 states:
“Documentation shall include records of management decisions, ensure that actions are traceable to management decisions and policies, and ensure that the recorded results are reproducible. It is important to demonstrate the relationship from the selected controls back to the results of the risk assessment and risk treatment process, and subsequently back to the ISMS policy and objectives.”
Test case versioning ensures that you are able to retain full historical records about test activities that were carried out and demonstrate full traceability from development to testing to release.
As is often the case, what works in your scenario might be different, but a couple of core tools will help provide a starting point to bring test versioning into your organization:
Version Control Systems
Code versioning is a widely accepted way of working in software development, and Git is a common tool to accomplish it. Two large suppliers of this are Github and Gitlab but other options are also available. Within Git, every change you make to a test written in code is committed and its powerful tooling allows you to tag particular branches or commits with version numbers.
Similar to version control for code, test case versioning could also mean: C comparing test case versions side by side, reverting to previous versions of test cases, and creating multiple branches and test case copies within a project.
Working closely with the development team will allow your test version numbers to align with the software version. So version 1.4.2 of your test suite is the suite of tests that version 1.4.3 of the software release is tested against.
Test Case Management: How to version your test cases
Although you can version tests written in an excel file or google sheet by creating separate tabs or spreadsheets, finding historical test case data is a messy task and with other tools that don’t support test case versioning, you would most likely be manually sifting through heaps of past versions trying to figure things out for yourself. It’s not an ideal situation.
Test case management tools like TestRail allow you to manage your test cases in a unified platform. Yes, you can use test management tools to write test cases, but certain tools like TestRail automatically track changes to test cases over time across multiple versions.
Image: Test case versioning in TestRail Enterprise allows users to compare any two test case versions side-by-side and see highlighted differences between the two versions.
TestRail gives users the ability to make faster decisions and more quickly analyze the changes that occurred over multiple versions by highlighting the differences in versions and allowing for easy comparison and giving users the ability to better understand how to fix issues, why a bug is occurring, and why something is not working the way it’s supposed to work.
As software projects evolve, every testing team needs to think about how they will handle changing requirements and test cases for their projects and releases over time.
Benefits of versioning your test cases
As the application evolves, so will your understanding of the scenarios, and your testing coverage will change along with it. By retaining historic information you can be better informed when selecting test cases to form part of the test suite or smoke tests.
For example, if you write a test because you want to ensure a certain behavior implicit in part of that test, and that test is changed, then that implicit non-functional requirement (NFR) might be lost with it, intentionally or not. If information is lost, future decisions will not take it into account and could lead to unintended consequences or tests not behaving in the way they were designed or expected to perform.
Another scenario to consider is that some industries require documented test reports of the test carried out at the time of the release (this is particularly true in industries such as Government, Health, and Banking). Versioning allows you to use tools to “go back in time” to see exactly what tests were carried out for a particular release and generate reports. What tests were run as part of Release 1.1 back in 2020? Were any skipped or failed? Did they test scenario X or Y? Test case versioning helps you answer these questions.
Conclusion
So why is test case versioning important? Because it allows testers and the rest of the team to understand how tests have evolved and changed during the lifecycle of a project. Having the historical information available helps testers make informed decisions as to where to spend their (often limited) time and ensure the quality of their work. Just as importantly, when a new tester joins the team they can easily see the decision process as to why a test has evolved over time, or indeed if it is no longer required. This will help them get up to speed with a system more efficiently and mitigate “tribal knowledge” stored only in the minds of team members. Arguably most importantly, it allows you, as a tester, to answer with confidence when asked “what did you test?”.
