Reports Webinars Questions and Answers

During the Winning at TestRail Reporting webinar, we promised to try and answer as many of your questions about reports as we could. Where possible, we did that during the webinar itself using the GoToWebinar console. But predictably, there were a great many that we just didn’t have time to answer on the day.

We did gather them all up to be answered at a later stage though. And so, for your TestRail reporting edification, here they are – with our answers to them – below.

TestRail Reporting Q & A’s

“Running tests” means manually executing these tests and input the result back on to TestRail?

Yes, although running tests could also mean running them externally by means of some form of automation (Selenium, or unit tests for example) and then updating TestRail with results via the TestRail API.

How could you create new reports to be added on reprots tab?

New reports can be added by clicking the Plus icon next to the desired report template. If you want to add your own custom report, please remember that this option is only available for Server customers currently, and follow the instructions in our guide here: http://docs.gurock.com/testrail-custom/reports-plugins-custom-1

How can you report from test cases?

There are many ways to report on test cases, depending on what it is specifically that you want to report on. For example if you wish to see the change or run history of a test case, you can view that directly within the test case by navigating to the History or Tests & Results sections accordingly. Alternatively, if you are interested in changes over e.g. all of the test cases in your project, you can generate an Activity Summary report for the desired timespan, which will then provide information on all of the (in scope, depending on the configuration) changes for the test cases (or sections, or suites – again depending on the configuration) within the project.

Any plans to add estimated forecasts at a test run level for the automation testing?

This is something we may look at for our automation features in TestRail 7.0.

How are we supposed to find test case results on JIRA if you cannot search for them since you do not support JQL?

You can do this in TestRail with the Cases > Coverage for References report. To do this, you would need to have a list of JIRA issues from JIRA (e.g. created via an export or query) and then create a new Cases > Coverage for References report with the “The following references only” option enabled. You can copy the list of JIRA issues to the text box below this option and the generated report would show the case coverage for your JIRA issues (including those issues without cases).

Is there a way to customize the body of the email within TestRail, prior to sending the PDF when sharing?

If you’re using the Server edition, you can customise the email text by modifying the lang files per instructions here: http://docs.gurock.com/testrail-custom/lang-translation. If you’re on the Cloud version, then I’m afraid we don’t currently provide a mechanism for customising email text.

Will the [Reference] link work if you are using TFS?

TFS integration instructions for TestRail can be found here: https://www.gurock.com/testrail/tfs-test-management

How do you remove weekend dates so number of executions don’t dip to zero?

It’s not currently possible to do that on the tab charts (e.g. Activity – which is presumably the one you’re referring to). It’s something we’ll certainly look at for any Reports changes, currently scoped for TestRail 7.0.

On the main dashboard graph, when I have many projects can I choose only a few?

Not presently. Again this is something we’ll look at for UI enhancements in TestRail 7.0.

Can “Groups” be used as the “person” to send a report to?

Sending reports to groups isn’t currently supported.

We use two different ‘pass’ statuses based on whether or not it was tested manually, or the result is from an automated run. Is it possible to change which status is used to generate that % number? We’re using testrail cloud…

Not currently, but additional configuration for the activity graphs is certainly something we can look at as an enhancement when we do some more work on Reports (TestRail 7.0).

Can I get reporting on average time to complete a specific type of test run that we run regularly across milestones? (i.e. QA environment tests)

This thread goes some way to answering your question I think https://discuss.gurock.com/t/how-to-created-a-report-for-execution-time-for-each-sectionid-in-milestone/3288 but if you have some additional questions or thoughts please do let us know and we’ll see what can be implemented in this regard in a future release.

Do the todo lists or the test run pages refresh, because it seems like you have to manually refresh them. I’ve had team members double up on testing because they didn’t know someone else had already tested a case.

You’ll need to manually refresh the pages in order to see changes made by other team members.

In defects tab, how do you reflect the issue status from jira?

Hovering over the defect reference will display a dialogue that contains the issue status from Jira.

Can you filter by test result?

Cases > Activity Summary, Property Distribution and Status Tops all allow you to either group or filter by test result status.

Is there any way to get the report based upon any customized fields? 

Custom fields can be included in various reports, however it depends on the data you’re attempting to grab. You can add custom fields as columns to any of the Tests or Test Cases tabs of the reports that support this option, as long as it’s not a long form field. You can also compare the values of any custom field with predefined options (e.g. such as Dropdown fields) with the Property Distribution report. For any specific use case data you’re attempting to retrieve, feel free to email [email protected] and we can let you know if this is something that our built-in reporting offers.

Is there a way to add to TestRail reports, where we can integrate TestRail with TeamCity, to keep track of pass-fail results of automated tests that link with the TestRail Test Cases/Run? In regards to regular build and running of automated tests as part of CI/CD?

There’s no direct integration with TeamCity currently. Typically, teams will use the TestRail API for this purpose, implementing it as part of their test automation framework and using it to report results back to TestRail in whichever way is preferred depending on the specifics of the project.

Some third-party vendors have developed plugins for this purpose, e.g. this one from AgileTestWare https://agiletestware.com/docs/pangolin-docs/en/latest/ci-integration/teamcity/.

Final Notes

If your answer didn’t make it into the list above – please accept our apologies. As you might imagine, many of the questions were quite similar to each other so we endeavored to cover as much ground with the questions and answers as possible. If you can’t find precisely the answer you’re looking for above, please do feel free to get in contact directly via [email protected].

And finally, thanks for all your survey responses after the webinar. Please rest assured that we’re taking your feedback and suggestions on board as we continue to improve and enhance TestRail over the coming months. Please stay tuned for future webinars and surveys for more opportunities to tell us where and how you think we can improve.

Post by Simon Knight, Gurock Product Manager. Follow him on Twitter or LinkedIn.