How Eventbrite Built and Scaled their QA Team

Learn how Eventbrite standardized, streamlined, and scaled their software testing and QA operations as they grew from a high-growth startup to publicly-traded company.
Industry
Event Technology
Location
USA

Founded in 2006, Eventbrite has come to support more than 949K organizations and creators around the world, handling more than 309M tickets for 4.7M+ events in over 180 countries in 2019. Eventbrite’s laser-focus on delivering quality to customers has been a key driver of their success, which culminated in their IPO on September 20th, 2018.

The Challenge

When the QA team at Eventbrite started their free trial of TestRail in early 2016, the company had already entered the hyper-growth phase that would propel them to their stock market listing two years later. They were in a position similar to many QA teams at high-growth startups: managing their test cases through a series of cobbled-together spreadsheets.

As Eventbrite SQA Engineering Manager, Andrew Smelser, recalls, “The start of QA at Eventbrite was from the grassroots. We began with just writing tests down in a free tool like Google Sheets.” 

The problem was that as Eventbrite exponentially grew, using spreadsheets to test became a bottleneck and a liability. Spreadsheets were fast and flexible, but they didn’t scale. They were hard to organize, were difficult to report on, and prone to error.

The Solution

Instead, the QA engineering team needed to find a solution that could:

  1. Help accelerate release cycles without sacrificing quality as the pressure to develop new features—and potential risk of new defects—was only going to continue increasing.

  2. Scale their ability to support a constantly growing engineering department, standardize tracking and reporting around key quality metrics, and help maintain clear, concise, and meaningful communication between disparate departments.

  3. Offer a competitive price point and flexible workflow that allowed team members to onboard quickly as the team scaled up or down in response to the needs and circumstances of the business.

“At the business level, we’re shifting the culture of ownership around quality. It is a big part of our agile initiative: that quality belongs to everyone.”
Andrew Smelser, SQA Engineer, Eventbrite

Andrew Smelser

SQA Engineer

Visibility

According to Eventbrite’s SQA Engineering Manager, Andrew Smelser, “As Eventbrite grew to the point where it needed a dedicated QA tool, we turned to TestRail.”

The first step was to centralize testing efforts. “We recommended that teams use TestRail because we could have all the information together in one easy-to-access place,” explained QA Engineer Diego Girotti. 

Using TestRail as a hub improved visibility into testing efforts, as well as traceability and auditability of testing history. TestRail also allowed those outside of QA and development greater insight into Eventbrite’s testing work and key business metrics such as test coverage, regression testing percentages, and automation.

Smelser summarized: “You gain a lot of visibility when you have this platform versus doing it in sheets with varying permissions.”

Flexibility

One of the most significant advantages of TestRail over spreadsheets was its flexibility and intuitive workflow. Even though some of the software developers on the team were resistant to using a new tool, TestRail’s ease of use won them over, particularly with the ability to import tests from external sources.

According to Smelser, “TestRail is not prescriptive,” and does not demand one rigid method of use. Individual teams at Eventbrite use TestRail in the way that suits them best, while still managing to track and store all of their test artifacts in a single tool.

This freedom was a massive benefit to Eventbrite. Utilizing TestRail did not necessarily mean replacing previous workflows entirely; rather, TestRail provided an extension of those workflows and helped to make them more efficient.

Learn how TestRail approaches test management

Compatibility with Test Automation

Eventbrite’s automation stack has evolved a lot since their early days as a startup, moving between multiple tools such as Selenium and Cypress over time. Because TestRail is a standalone test management platform that integrates with external tools via an API, it doesn’t lock teams into specific tech stacks or vendors.

For the Eventbrite team, this meant that their overall test management strategy has remained stable, consistent, and dependable, despite changing tools and environments. As Eventbrite moves forward, TestRail provides the stability to support company initiatives like increasing test automation and implementing a CI/CD pipeline.
Customers want to know if their projects are ready to ship and TestRail allows the team at Cardinal Peak to demonstrate –with confidence- that they have coverage of the project’s unique requirements and that the product has passed all tests. 

Pricing

TestRail’s flexibility and ease of use even extends to TestRail’s pricing model. TestRail uses a balance-based billing model where you are billed on the first business day of the month for product usage in the previous month, based on the maximum number of active users you had in your account that month. Users can be activated and deactivated as needed without any contracts or fixed terms, and monthly deductions/charges adjust accordingly.

For Eventbrite, this meant that the team could scale quickly without having to slog through lengthy pricing or contract negotiations. They were never “locked-in” to any user count; they paid for what they used, and as new business challenges arose, they could adjust quickly.

According to Smelser, “By being able to prepay, and then deduct monthly charges against that balance based on how many seats we actually used, we’ve been able to stretch our contract a couple of extra months. And obviously, the opposite has been true for us too: we’ve eaten our balance quickly sometimes because we grew so quickly. But we’re really happy with that flexibility.”
“As Eventbrite QA engineering has continued to evolve, TestRail has been a constant part of our tech stack.”
Andrew Smelser, SQA Engineer, Eventbrite

Andrew Smelser

SQA Engineer

Shift in the Culture around Quality

While visibility and flexibility were significant benefits for the Eventbrite team, the real value of TestRail came from the fact that adopting TestRail helped the team successfully change the culture around quality across the company. 

“At the business level, we’re shifting the culture of ownership around quality,” explained Smelser. “It is a big part of our agile initiative: that quality belongs to everyone. The ideal is that all development teams own quality.” 

This movement — for quality to be owned by every developer, and not strictly by QA — took shape in a push to “shift left” testing across the board. “The goal is to be testing earlier and more frequently,” explained Smelser. And at the center of that shift was TestRail.

TestRail empowered the Eventbrite QA team to collaborate with Product and Engineering to scope new sprints and build quality into the development process from the outset. Getting developers involved in testing also helped focus their development on real outcomes for end users, leading to better code, faster releases, and increased value to the customer

Eventbrite HQ
The Eventbrite HQ before the Covid-19 pandemic. Source: Eventbrite.

This new paradigm shifted the role of the QA members within the organization. It allowed the QA team to contribute at a larger scale through the formation of an “on-demand” testing team that could assist with the quality assurance needs of multiple teams across the organization. If certain functional areas or integrations seemed to be taking too long to test, failing repeatedly, or resulting in higher numbers of customer-reported incidents over time, the on-demand team could “jump into TestRail and see what [the team was] testing,” according to Smelser, and advise on different strategies to make improvements. 

Finally, by standardizing test management in a centralized system and giving everyone on the team access, the Eventbrite QA team was able to better promote their goal of constant improvement around testing and QA. By reviewing metrics around the quantity and severity of defects identified across various sprints using TestRail, the team could estimate how long it would take to develop, test, and ship high-quality releases more accurately. 

They were also able to include more cross-functional stakeholders in conversations around quality more easily, connecting the dots between product, engineering, QA, and support to analyze the impact of new changes on the overall quality of Eventbrite’s ecosystem of apps.

The Results

TestRail’s flexibility and usability allowed cross-functional team members across Eventbrite to quickly onboard new users and integrate new development tools while maintaining a stable but fast-paced testing environment. 

Tracking all manual and automated testing operations through one central platform increased visibility into testing as a whole, and helped the team monitor the quality of their products with each new release. 

This visibility then shaped business-level conversations around Eventbrite’s overall software development and delivery process itself, facilitating a culture shift and driving more meaningful adoption of agile processes across the team. Getting the whole team involved in testing and QA allowed Eventbrite to scale more efficiently while constantly improving the quality of the product for end-users (even as the Eventbrite’s TestRail user count scaled more than 400% in less than four years).

Ultimately, TestRail continues to be the right test management solution for Eventbrite for three reasons:

  1. TestRail has been able to scale with Eventbrite’s test management needs, from high-growth start-up to publicly-traded company.

  2. TestRail provides a balance between features and usability that makes it easy to use for Eventbrite’s QA & Engineering teams.

  3. TestRail has a flexible, competitive pricing model that works in Eventbrite’s favor. 

“You don’t have to beat TestRail by an inch. You would have to beat TestRail by a mile. The pricing model is competitive enough where, to outweigh the cost of making a transition, a tool would have to be significantly cheaper and/or offer additional functionality.”
Andrew Smelser, SQA Engineer, Eventbrite

Andrew Smelser

SQA Engineer

Join over 10,000 QA teams using TestRail to release flawless products, faster

3D Systems

Read how TestRail’s reporting capabilities and ease of use provided the QA team at 3D Systems a way to improve organizational communication and streamline their test management processes to deliver the highest quality 3D printing products.

Solitea

Learn how TestRail’s ease of use, integrations, and reporting capabilities provide the QA team at Solitea with the necessary solutions to find bugs faster and accomplish more testing in less time.

Convercent

Learn how TestRail has given the Convercent team a scalable solution that supports their growing QA team and helped them standardize their test management in a centralized system, maintain meaningful communication between departments, and get higher-quality releases out the door faster.

Build quality processes and release with confidence