If you are involved in software testing and use tools like TestRail or Selenium, it’s really important to be able to answer questions like: “What’s the status of testing? How much testing is left? Do we feel confident that we’ve done enough testing to ship this release?”
To make sure that you have complete visibility across all of your testing activities, it’s crucial to report test results from your automated testing as well as from any human-centric (or manual) test activities. Sending test automation results to TestRail gives you the whole picture so that you can quickly and accurately answer those questions that will help guide decisions and track overall product quality over time.
TestRail has a RESTful API that can be used to integrate with various tools, frameworks, and third-party applications like test automation frameworks. This document demonstrates a similar use case where teams can integrate their automated tests with TestRail. You will learn:
- How to set up your TestRail account to work with test automation, in this case, Selenium UI automation.
- How to parse the results from automated tests executed outside of TestRail (e.g., as part of a continuous integration system like Jenkins) and how to use TestRail’s API to submit the test results.
- How to trigger a test automation run from the TestRail application with UI scripts.
This demonstration uses the TestRail Cloud account. This blog assumes that the reader has a basic understanding of Selenium, C#, Jenkins, and GitHub and will be helpful for testing professionals responsible for carrying out automation activities.
The basic workflow for integrating Selenium with TestRail
To integrate TestRail with your Selenium UI automation, you can use the TestRail API. The basic flow is as follows:
- Run UI automation.
- Parse the Selenium results file.
- Post the results to your TestRail test run using the TestRail API.
Because every team’s implementation of Selenium is unique, you will need to add a function to your Selenium UI automation project to parse the attributes from the Selenium results file that you want to report in TestRail.
Teams typically post test automation results to TestRail using one of two methods:
Method 1 – Single Test Case in TestRail:
- Add a single “Selenium Test Automation” test case to your TestRail project.
- Create a new test run in your TestRail project and include your “Selenium Test Automation” test case.
- After running your automation tests, parse the results file and update the TestRail “Selenium Test Automation” test status to “Passed” or “Failed” based on the overall results from your automation.
- Then add the results for individual automated tests as a comment
Method 2 – Multiple Test Cases in TestRail:
- Add test cases in TestRail for each of the tests you’ve automated with Selenium.
- Create a new test run in TestRail and include all of the test cases that will be tested when you run your Selenium automation tests.
- After automation tests are complete, parse the results file and update the status of each test in the TestRail test run based on the results from your automation.
- You can add additional comments or results for individual steps as a comment for each test as needed!
Method 1 makes it faster to report test results to TestRail from your test automation but doesn’t give you the same level of granularity in the status or historical results for each of the tests you’ve automated. For that reason, Method 2 is the most common way teams integrate TestRail with test automation frameworks.
Triggering test automation from TestRail:
One of the handiest advanced features of TestRail is the ability to add custom UI features to your TestRail projects, such as buttons, links, and other UI elements. You can script what happens when users interact with these custom UI scripts—such as triggering test automation to run outside of TestRail.
The diagram below illustrates the basic flow for the integration we will be setting up in this guide. It starts with using a UI script in a new test run we’ve created in TestRail to trigger a Selenium test automation to run and post results back to TestRail.
Key Concepts
1. UI script
There are boilerplate templates for the UI scripts available on the Gurock public GitHub repo. More specifically, for this blog, we used this UI script template and slightly modified it to add Jenkins details.
This modified UI script adds a new button to the TestRail toolbar for test runs. When clicked, it sends an HTTP request to the Jenkins server, which in turn, triggers the automated tests.
Please note that most teams will trigger test automation as a regular part of the CI/CD pipeline. The UI script is a good way to trigger automation if you want to test outside of the regular CI/CD pipeline workflow.
2. UI automation using Selenium
Here, the selenium UI automation is written using C#.Net for demonstration purposes. This test automation has been set up to run as part of a Jenkins job.
3. Test automation execution and result parsing
Once you click on the “Start Tests” button from a test run, Jenkins triggers the automation pipeline. After automation execution completion, the result is stored in the Jenkins agent in XML file format. I am parsing the result file and retrieving the test case title, test outcome (passed/failed), and failure reason (if any).
4. Update results back to test cases in TestRail
Once the automation execution results are available, I use TestRail’s add_results_for_cases API, which bulk-adds multiple test results in one step.
Step by step guide for the configuration and execution of test cases
1. Create test cases in TestRail
Here we have created four test cases in TestRail, as shown in the screenshot below. The three test cases (highlighted in red) have automation types set to Selenium (highlighted in purple) and have corresponding automation code in the UI automation project.
2. Add a ‘Start Tests’ button using UI script
UI scripts are an advanced feature in TestRail that allows you to add custom buttons, messages, or other display elements within your TestRail instance. In this case, we will add a new button to the test run toolbar to trigger a test automation run. When clicked, it sends an HTTP request to the Jenkins CI server, which in turn triggers the automated tests.
If you have administrative privileges in your TestRail instances, you can add the UI script (trigger. ui) in TestRail’s administration area under Administration > Customizations > Add UI Script. Simply copy and paste the whole UI script from the GitHub repository to the Configuration text box. After adding the script, you should see a new Start Tests button in the toolbar for test runs:
Script snippet:
This script is invoking the Jenkins build jobs URL by passing the current run id as a parameter. You can get a run ID of the current run using uiscripts.context.run.id. We need this run ID later to update the test results back in TestRail.
3. Create a test run
Add a new test run in TestRail, selecting the test cases created in Step 1, which have corresponding automation test code. Once UI scripts are added, we can see the ‘Start Tests’ button in the toolbar for all the test runs.
4. Set up and run UI test automation using C# & Selenium
In this demonstration, the UI test cases are automated using Selenium and C#.Net. The system under test for UI automation is a personal TestRail cloud instance.
If you want to follow along with the actual code used in this demonstration, the UI test automation project can be found in the TestAutomationProject solution in the GitHub repository.
A point to note here is that the automated test case name and title of test cases in TestRail are kept exactly the same. The title is mapped between UI automation (as shown in the screenshot of our Visual Studio project below) and the TestRail test case, which will be used while updating the test case results back to TestRail. While this is not strictly required, it makes it much easier for you to reference your test automation code and TestRail test results later.
Test automation file in Visual Studio:
Test case titles in TestRail:
5. Parse the results and update back to TestRail
Once the test automation execution is completed, the result file will be generated in the location specified in the Jenkins test automation execution step. For me, it’s the C directory (C:Automationresult.xml) in Jenkins agent.
We have written a second script to parse the automation test result file and retrieve the test name, outcome, and error information for the failed test case (highlighted in the screenshot of our sample test result file below).
Sample XML test result file:
a. Code to parse and store needed information:
This code loads the result.xml file and retrieves the necessary information from each test result’s attributes like outcome, failed test’s error logs, and testName, and then, stores the information into the testResults object.
Now, we need to get the test case details from test runs, for which we need to update results back to TestRail. Here, we get test case details by calling the get_tests API of TestRail.
Because we want to call TestRail’s API, we can use API binding to access TestRail’s API from . NET. Please refer to .NET API binding for more details.
In the example below, we are retrieving the test case title and test case id of the test cases present under a particular test run and storing them in a dictionary data structure.
So, dictionary contains { “test name”, “case id”} key/value pairs
e.g. TestDetails(key, value) =>
{“Login_With_Valid_User”, “3281”},
{“Login_With_Invalid_User”, “3282”},
{“Demo_Failed_Test_Case”, “3283”}
b. Code to get test case details:
Now that we have the data of test cases from test runs and test results for those test cases, we can update the results back to TestRail in bulk using the add_results_for_cases endpoint.
If the test case title and test case name from automation match, then update the same test case outcome in TestRail. Furthermore, for failed test cases, we made sure to update the test case comments with failure reasons.
c. Code to update test result in TestRail:
Updated result in TestRail:
After you add the functions to parse your Selenium test results file and post the results to TestRail, you can test the integration by running the automation project!
Now, we can see a rich report on the status of our automated tests, including seeing which test failed and comments from the .xml results file indicating where the test failed. Test case with automation type as the manual was not executed as it didn’t have corresponding automation implementation.
You can find parsing of the test result and update the test result back to TestRail code in the TestRail.Integration solution at the GitHub repository.
6. Jenkins integration
As the final step of this guide, here is a brief overview of how you can set up a CI/CD tool like Jenkins to run your Selenium automation as part of your build pipelines. The specific setup of your build pipelines will depend a lot on your team’s build and deploy pipeline; check out the screenshots below demonstrate how I configured the Jenkins job to run in this particular demo.
Step 1: Create Jenkins project and configure source code management by providing Gurock Repository URL.
Step 2: Create build trigger and copy the same in UI script to invoke Jenkins CI system (refer to the section above).
Step 3: Compile the test automation project.
Step 4: Run the test automation project, which executes UI automation.
Step 5: Compile the TestRail integration project which creates TestRailIntegration.exe which we are using in the next step.
Step 6: Run the TestRail integration project, which internally parses the test result generated from the above steps and updates the test result back in TestRail.
For more information about the topics covered in this guide, you can view the resources below.
Resources:
