We are excited to share the insights from the 2020 TestRail User Survey. We are listening and want to share the results to help us all better understand the community that makes up TestRail.
In total, we received over 2,500 responses to this year’s survey (2,549 to be exact)! To break it down, here are some key demographic highlights from this year’s survey.
- Most people who answered the survey were testers or QA engineers, followed by test leads and test automation engineers (SDETs)
- They represented 30+ industries, with the largest being computer software
- Survey participants represented almost 100 countries around the world
- 80% of those who took the survey use TestRail multiple times per week—50% use TestRail every day!
Let’s dive into the details.
A bit about you, our community
Our survey included questions to get a better sense of who we were hearing from. Here is some of what we learned.
Your role and how we fit in
What job responsibilities and/or titles do you hold? Based on the participants in this year’s survey, there is a good chance you fall into the category of a Test / QA Engineer (48%) or a Test Lead/ QA Lead / Manager / Director (22%).
How large is your QA/ testing team? 56% of you have teams of 1-10 testers and 28% have teams of 11-50 testers.
What industry does your organization work in? Computer software, Telecommunications, Technology, Internet & Electronics ranks highest with 51% and the rest were a mix between various industries.
Your usage of TestRail
When asked how long you’ve been using TestRail? 44% said that you have been using TestRail for 1-3 years, while 29% said that you’ve been using TestRail for less than 1 year (Welcome to those folks who have been using TR for less than a year!).
When asked how often you use TestRail? 49% answered that you use Testrail every day, while 29% answered that you use TestRail multiple times a week.
Testing types, techniques, and/or development methodologies your team currently uses
About 82% of you have embraced the Agile testing methodology and 65% use Scrum. This shows us that the industry is shifting towards faster development and release pipelines. The growing popularity of these testing methodologies shows us the demand being placed on QA teams for faster delivery cycles and reflects the need for test management tools to be an essential part of the quality management process. The reality of today’s development world is that fast turnaround and overall development efficiency are of utmost importance, and the test management you use for agile testing must allow you and your team to meet your quality goals and complete testing on time.
Metrics or KPIs your team report on
In terms of the key KPIs you and your teams use to measure your testing, over 50% of you said that your teams track KPIs related to testing activities such as testing progress (were pass/fail/blocked %), the percentage of their tests that are automated vs. manual, test coverage, total number of tests executed, and the number of defects reported during testing. A smaller percentage of participants reported tracking quality metrics around code that has been deployed to production, such as production incidences or test effectiveness (% bugs found during testing vs. production).
Percentage of automated vs. manual tests
35% of your tests are automated. When asked, “What percentage of your tests do you expect to automate in the next year?” you answered that you expect to be automating around 54% of your tests by the next year.
This tells us how important it is to have a comprehensive test case management system in place. This data point resonates with conversations we’ve had with you in TestRail Community forums, interviews and through webinar Q&As over the last year or two: QA engineers are under ever-increasing pressure to do more testing in less time, and test automation promises to help take some of the most repetitive tasks off your plate so that you can focus your efforts around QA craft management (quality risk analysis, test design and planning, and analyzing data that comes back from testing).
These data from the survey also highlight why it’s so important to make sure you have a test management solution in place as you automate more of your test suite: test management systems give you a way to keep track of which tests have already been automated and which are still manual, help you highlight opportunities for further automation, and help maintain consistency around reporting on test progress and results across all of your human-centric and robot-driven testing activities.
Why do you use TestRail?
When asked about the primary reasons why you and your team use TestRail, a whopping 65% of you answered that one of the reasons is to maintain a repository of reusable test cases, 63% reported that you use TestRail to document test plans, and 59% said that it was to make it easier to track the testing status. Moreover, 46% of you reported that you use TestRail to maintain a record of historical test results, while 42% said that you use the tool to both be more productive/efficient during testing and to get more visibility into the testing process.
This tells us how TestRail plugs into your overall QA process. With software development evolving rapidly, companies need to look for ways to optimize testing processes and improve speed without compromising quality. From allowing you to maintain a repository of reusable test cases, to giving you more visibility into the testing process, TestRail is used to handle the complete lifecycle of software testing and empowers you with the visibility, flexibility, and insights required to increase efficiency in your testing processes.
Our genuine thanks go out to those who took the time to answer the survey. It is immeasurably valuable to us, and we’re looking forward to applying it to all the amazing TestRail releases we have coming your way in 2021.
We’ve learned so much, it doesn’t fit into one post. Stay tuned for Part 2: Your tech stack, your biggest challenges & your goals for the year.
Thank you again to everyone who participated in this year’s user survey!
It’s with great gratitude that we share the list of individuals who were picked in the prize drawing to receive a €25 or €100 Amazon Gift Card (who have permitted us to share their first names). To everyone below and the 13 other individuals who asked us not to share any of their personal information, congratulations and thank you again for participating!
Amanda
Donovan
Daphne
Chris
Tim
Srilata
Nicola
David
Julia
Yumiko
Srilatha
Rick
Chetan
Tatyana
Nayeli
Carleen
Steven
Martina
Emily
Catherine
If you didn’t win this time, keep your eye out for more chances to participate in the TestRail community and potentially win other prizes over the coming year!
