We’re excited to announce the general availability of TestRail 7.5 for TestRail Cloud and Server customers.
TestRail 7.5 introduces webhooks, which give you the ability to trigger real-time notifications based on events in TestRail like saving new test cases, creating new test runs, adding test results, and more. This allows you to build custom integrations with other tools in your tech stack—like triggering CI builds, programmatically kicking off test automation, and getting notifications in communications channels like Slack—so that you can streamline the connection between TestRail and the rest of your software development toolchain and increase visibility into testing activities for everyone on your team.
TestRail 7.5 Server is now in full parity with features available in TestRail Cloud 7.5, including security improvements with expanded SSO implementation standards (available on the Enterprise edition), Multi-Factor Authentication (MFA), and new API endpoints for enhanced User management.
Overview
At the heart of the new webhooks functionality is the ability to automatically send notifications and payloads to external systems or applications when events occur in TestRail, like when new test cases are added, test runs are created, or test results get updated. This way, you can automatically trigger an automated test build or pipeline in your CI tool through its API or post real-time updates to a Slack channel.
With webhooks, you can now integrate TestRail with the rest of your tech stack, save time, and give everyone on your team more visibility into the activity and progress of your testing. You can create a webhook globally across all projects or choose specific projects that you want to apply your new workflow to.
For TestRail Enterprise Server customers, we’ve expanded Single Sign On (SSO) functionality by adding support for OpenID Connect (OIDC) and OAuth 2.0 standards, adding to TestRails existing support for SAML 2.0. Now more organizations can enforce SSO policies to help ensure users’ credentials aren’t lost, stolen, or reused.
Speaking of keeping data safe within your TestRail instance, all TestRail Server customers can now make sure users log in with Multi-Factor Authentication (MFA). By enforcing authentication with email and popular authenticator apps like Google Authenticator, Microsoft Authenticator, and Duo mobile, you’re better protected from stolen credentials being used to access sensitive data.
That’s not all! We just made it easier to create users and user groups with your LDAP service, like Azure Active Directory, with new API endpoints. You can also now control which projects users and user groups can access and retrieve and update additional user information, such as group membership and administrator access. As an added layer of security for the TestRail API, administrators now have the option to require the use of API keys instead of account passwords.
New Webhooks management console
Head over to the TestRail Administration console and navigate to the Integrations tab. From there, choose the Webhooks tab. The new webhooks management console makes it easy to create new webhooks as well as test, monitor, and manage webhooks you have already created.
Monitoring and managing existing webhooks
In this example, we have created two webhooks already. The first one sends a notification to Travis CI that triggers a CI/CD server build job that runs a suite of automated tests. The second one sends notifications to Slack when new test cases are designed to help communicate to teams as our test coverage grows for maximum visibility into our testing efforts.
Good news! The status column shows that both of our webhooks are currently working because it is receiving a HTTP response code of 2-something. If the webhook receives a different response, like a 404 or a 500, the status for the webhook would be flagged red, indicating we need to investigate the problem.
Each webhooks has a name and some additional information relating to the TestRail projects with which they are associated with as well as the events which will spawn a webhook payload. This way it’s easy to parse through your list and recognize each one.
Need to update a webhook, deactivate it for the time being, or delete it? You can do that under the actions column on the right of the webhook that’s misbehaving or has reached the end of its days.
But how did we set up these sweet, sweet webhooks? Let’s take a look!
Customize your webhooks
To create a new webhook, click on Add Webhook. Here’s one of the webhooks you saw above, which triggers a build in Travis CI:
To create this webhook we:
- Gave it a (relatively) descriptive name.
- Provided a Payload URL. This is where the webhook content will be sent to when it is triggered.
- Set a Method and Content Type indicating whether we’re POSTing, GETing, or using some other method. We also indicated what the content type should be.
- Set some default headers (those are the ones in gray) as well as some custom headers (those are in black).
As you can see, our webhook doesn’t have much of a payload yet. We’ll show you how to prepare that below. Travis-CI happens to not require much in the way of a payload, hence it’s empty in this example.
On the right, you can see our recent webhook deliveries displayed, providing some visibility to what our webhook’s been up to lately. The Test button at the top right of this section lets you test your webhook configuration to make sure it’s behaving the way you want it to.
In the settings further down the page:
- You can add a Secret to your webhook if you need one. This will be included in the webhook headers.
- You can select which TestRail events will trigger the webhook and send content to the corresponding Payload URL. The event names should be self-explanatory, but if you select the Plan created event, for example, the webhook will send a payload when a plan is created in the scope of the selected projects.
- The projects for which the events should be applied can be selected in the Projects area. You can see in this example that we only have one project in our instance so far and it has been selected.
- You can activate or deactivate the webhook by checking (or unchecking) the Active checkbox.
Once you’ve got your webhook all configured, you can save it, test it, and leave it to run per your configuration.
In the example I’ve provided so far, my webhook payload has mostly been empty (apart from the headers and URL content). Depending on your needs though, you may need to define some payload content – so let’s take a look at that.
Customize webhooks payloads
The webhook payload field is designed to be flexible so that you can construct and send more or less any kind of payload you wish, using the webhooks console.
Example use cases for TestRail Webhooks
Because you can customize webhooks and webhooks payloads from TestRail just about any way you would like, there is an almost endless number of ways you can use webhooks to connect TestRail with other tools in your tech stack. That said, two of the most common use cases we anticipate TestRail users to adopt are one, using webhooks to trigger CI/CD build jobs for test automation and two, posting updates to communications tools like Slack or Microsoft Teams.
Trigger builds for test automation runs in TravisCI
A common use case for webhooks is using them to trigger a CI/CD server build job, which may run a suite of automated tests, using Jenkins or, in this example, TravisCI.
The TravisCI documentation states that all you need to do to trigger a build is to use their API and ensure your request includes the relevant repository and job details, along with the appropriate authentication:
This simple example will have the effect of triggering the build job and executing it without any further instructions. More complicated scenarios can be accomplished by adding JSON formatted instructions to the webhook payload.
For additional scenarios and instructions related to your specific CI/CD build server, we recommend consulting the relevant documentation.
Trigger notifications to Slack
Another use case for webhooks might be using them to notify users of events in TestRail via a messaging app, like Slack or Microsoft Teams.
I set up a Slack app (instructions for doing so can be found here) to accept incoming webhook payloads from TestRail, and created a webhook in the TestRail webhooks console:
The payload field is a bit squished, so we’ve included the full JSON payload below to give you an idea of the kind of things you can do with webhooks. (Slack provides some further instructions for accomplishing different message formats here.)
| { “text”: “%event_creator% created a new test case:”, “blocks”: [ { “type”: “section”, “text”: { “type”: “mrkdwn”, “text”: “*%event_creator% created a new test case:*” } }, { “type”: “divider” }, { “type”: “section”, “text”: { “type”: “mrkdwn”, “text”: “*<%url%|%name%>* n Priority: %case_priority% n Type: %case_type%” } } ] } |
When the webhook is executed, you’ll end up with something resembling the below message from Slack:
As demonstrated in the example above, your payloads can be supplemented with data from TestRail, by using variables.
Use variables to get data from TestRail and populate them dynamically at execution time
We’ve made the following variables available to use in your webhook payloads:
- %assigned_to% = text, user name
- %case_priority% = text, priority assigned to test case, e.g. “High”
- %case_type% = text, e.g. “Automation”
- %completed_on% = timestamp
- %config% = text, runs.config
- %custom_x% = object/array, returns any fields associated with the field starting with “custom_”
- %description% = text, e.g. milestone.description
- %due_on% = timestamp
- %entity_created% = timestamp
- %entity_creator% = text, user name
- %estimate% = integer, estimate assigned to the test case (supplied by user)
- %event_created% = timestamp
- %event_creator% = text, user name
- %event_type% = text, {created, deleted, updated, assigned}
- %id% = integer, unique ID, e.g. case.id
- %is_deleted% = boolean, value indicating whether or not the case/run/plan/milestone has been marked as deleted
- %milestone_id% = integer, unique ID
- %more_info% = text, e.g. “”/api/v2/<endpoint>/<id>”
- %name% = text, entity name or title, e.g. runs.name or cases.title
- %project_id% = integer, unique ID
- %refs% = text, e.g. “TR-71, TR-72”
- %section_id% = integer, unique ID
- %stats% = object/array, {passed_count, blocked_count, untested_count, retest_count, failed_count, custom_status_n_count}
- %suite_id% = integer, unique ID
- %template_name% = text, template used by test case, e.g. “Exploratory Session”
- %url% = text, e.g: “https://sjpknight.testrail.com/index.php?/milestones/view/61”
- %secret% = text
The variables can be accessed by adding them to the payload field and will be populated with the corresponding TestRail data when the webhook is triggered:
The ability to add variables to webhook payloads is a beta feature at the moment and something we’ll be looking to improve on in a future release. If you have any feedback, suggestions (e.g. for additional variables), or requests for how to improve the functionality – we’d be very happy to hear from you about them!
On that note, there are a couple of known issues we are aware of.
Known issues
- The save webhook process is a little clunkier than we would like. We’ll be looking to improve the process as soon as we can.
- The payload field can’t be expanded at the moment, which makes life a little tricky when creating large payloads. We expect to improve upon this also. In the meantime, if you have a large payload, we recommend pasting it in from an IDE or similar tool.
- Webhook payloads aren’t triggered by API-related events, or some event paths in the UI (e.g. copying a case is equivalent to creating a case, but doesn’t trigger the add case event).
- The current list of variables that can be used in the payloads is more limited than we’d like. If you have any requests for more, please tell us about them.
If you spot any others, please do let us know by contacting us at [email protected]. Improvements are planned for an update release soon.
TestRail Server: New SSO Implementations
TestRail’s SSO feature allows administrators to integrate TestRail with their preferred SSO identity provider (IDP) using SAML 2.0, OAuth 2.0, and OpenID Connect protocols. In practice, this means that the management of users can be streamlined, by creating them once in the IDP and then providing them with access to whichever applications are required by the user – TestRail, in this case.
You can enable SSO by navigating to Administration > Site Settings > SSO, choosing your preferred protocol, and filling out the required settings. You can use the SSO configuration page to integrate with any identity provider that supports SAML 2.0, OAuth 2.0, or OpenID Connect.
While SSO is not a new feature in TestRail, support for OAuth 2.0 and OIDC is now available in 7.4, adding to the previous implementation of SAML 2.0. We’ve updated our existing documentation for configuring SSO and have guides for a variety of popular tools using each available protocol:
- OAuth 2.0 – Azure
- OAuth 2.0 – Google
- OAuth 2.0 – Okta
- OpenID Connect – Azure
- OpenID Connect – Google
- OpenID Connect – Okta
- SAML 2.0 – Azure
- SAML 2.0 – Google
- SAML 2.0 – Okta
Click here to learn more about Enterprise SSO implementation
TestRail Server: Multi-Factor Authentication
To enhance security for TestRail instances, we’ve added Multi-Factor Authentication (MFA) so you can ensure users authenticate their login via email or popular authenticator apps. This can be enabled for your TestRail instance and then enabled on a per-user basis. Any administrator can enable MFA for TestRail under Site Settings > Security:
To then require MFA for your users, you can enable this setting on a per-user basis or by editing your users in bulk. Requiring MFA will also require API keys for your TestRail users who utilize the API.
Once MFA is enabled, users will be required to enter a one-time use code when logging into TestRail:
By default, this code will be sent by email to the user:
Alternatively, users can connect their individual accounts to an authenticator app that supports the Time-Based One-Time Password algorithm, such as Authy, Google Authenticator, or Microsoft Authenticator. To configure an authenticator app, users should follow these steps:
- Navigate to My Settings > Auth and press the Connect button for an authenticator app:
2. Follow the instructions in the dialog:
- Save your user settings
Click here to learn more about MFA
TestRail Server: API Changes
The TestRail API received a number of updates to enhance permission management for individual users, user groups, and projects. User endpoints now return additional user details and project endpoints now return more permission information for users and groups. New endpoints have been added to allow for the creation and management of TestRail users and user groups.
Here’s a brief breakdown of updates to existing endpoints:
get_user and get_users support the following additional fields for users:
- email_notifications
- is_admin
- group_ids
- mfa_required
- assigned_projects
- sso_enabled
get_project, get_projects, and update_project now support the
following fields:
- default_role_id
- default_role
- groups*
- users*
The groups and user fields will contain user and group information specific to the project.
We’ve added the following endpoints to TestRail’s API to enhance user and group management
- add_user – create a new TestRail user
- update_user – update an existing TestRail user
- get_group – get information on a specific user group
- get_groups – get information on all user groups in TestRail
- add_group – create a new user group
- update_group – update an existing user group
- delete_group – delete a user group
- get_roles – get all available user roles in TestRail
Please Note: If Multi-Factor Authentication is required for users, this will require API keys for these users, regardless of the Site Setting enabled.
Getting TestRail 7.5
You can start a 14-day free trial of TestRail here (cloud or server): https://secure.gurock.com/customers/testrail/trial/
If you want to create a subscription for TestRail Cloud, you can do so from within TestRail via Administration > Subscription. Or, if you want to order TestRail Server licenses, you can do so from our website here: https://secure.gurock.com/customers/shop/annual/purchase/
Registered customers can download the full version from our customer portal: http://www.gurock.com/portal/
TestRail Enterprise
To learn more about SSO, Project Level Administration, or any of the other features on TestRail Enterprise, please email us for a trial or quote via [email protected], or you can use the contact form here: https://secure.gurock.com/customers/support/.
Not sure which TestRail plan you’re on? Reach out and we’d be happy to help.
Updating to TestRail 7.5
TestRail Cloud
TestRail Cloud instances are automatically updated to the latest version. You can check your version via the TestRail Help > About TestRail menu item.
TestRail Server
Registered customers can download the full version from our customer portal: http://www.gurock.com/portal/
