How to Structure, Perform, & Report On Exploratory Testing

Exploratory testing is a dynamic and unscripted approach to software testing where you test your application or product based on a particular goal or mission instead of a formalized script or set of test steps. 

Exploratory testing is particularly useful for uncovering unknown risks or deficiencies in your application — especially from a user-acceptance perspective — and it relies heavily on your testers’ domain knowledge, creativity, and intuition. 

The most common way to perform exploratory testing follows the five stages of session-based test management (also called the “SBTM Cycle”):

1. Define the mission or goal for testing 

2. Create a test charter

3. Timebox your testing

4. Review your results 

5. Debrief

How to structure exploratory tests 

Although exploratory testing is more spontaneous and adaptable than traditional scripted testing, it is structured in the step-by-step, time-based SBTM Cycle approach.

1. Set a mission or goal for testing

• Classify the bugs most commonly detected in similar software or past projects (you can categorize them by severity and priority).
• Identify, record, and study the root cause of the bugs
  • Collect details such as screenshots, comments, and reports and define the problem
  • Ask questions such as “What events led to the problem?” “What systems were involved?” “Who was involved?”
• Identify risks and create test scenarios to test the application (application testing)

2. Create a test charter

A test charter is a concise, high-level guide that acts as a mission statement that helps you focus your exploratory testing efforts and adapt to changing conditions while allowing room for creativity.

For example, if you are testing the web app login for an e-commerce platform, a test charter might look like “analyze homepage login menu functionality and report on areas of potential risk,” or “Analyze the login features of the web application,” or “Discover if people with false data can still log in to our web app.”

Charters are designed to help you stay on track and minimize a particular risk through oversight. When creating a test charter, you should include the following: 

• The main mission for your session
• The target area or features to test and how to test them
• The risks and assumptions involved
• The expected outcomes or deliverables and how the user might use the system
• What heuristics or tools you will use during your session

Example of an exploratory test charter

A charter for a social media application might, for example, be “Quick attacks around profile creation” or “Test multi-user comments, permissions, and race conditions within and out of group membership.”

A tester with the second charter would likely create three accounts, two of which are members of the same group, then send posts only that group can see, ensuring the third member cannot see them. The tester might make a post, see it show up for the second user, and then click delete in one browser and post the comment in the other two seconds later.

Typically, charters focus on a particular piece of the application, such as the shopping cart, checkout, the login/lost password process, or the reporting panel. However, they can also explore a specific user journey or new functionality. It is common for quick overview sessions when companies release a new browser or to check and see what it would take to support a browser or tablet. 

Using charters can take a vague, black box process and turn it into a collection of risks managed in this run.

3. Timebox your testing

A timebox is a test session where testers are focused and uninterrupted. 

Timeboxing involves the following steps during a predetermined amount of time:

• Testers work on the specified tests for an allotted amount of time without interruptions
• This is usually two testers working together for 60 to 90 minutes
• The timebox can be changed depending on test progression or as needed. This typically means the time is extended or reduced by 45 minutes

4. Review your results 

• Review and document your results
• Evaluate the bugs that you found and decide whether you need to perform any more investigations or if you have sufficient test evidence to report on the issues
• Analyze coverage areas

5. Debrief

An important part of session-based test management is learning from conducted test sessions and making testing accountable. To learn from your tests, doing a debrief of your test sessions together with another team member or stakeholder is a great way to improve future sessions, ensure you covered all important aspects of the system under tests and to review the effectiveness of your tests.

• Compile the results and outputs and compare the actual results to the expected results from your test charter.
  • Identify issues and risks
  • Identify opportunities for improvement
  • Document your results clearly
  • Communicate the test results with evidence and examples to support your claims.
• Decide if you need to test more. You can do this with the help of a well-written test summary report.

Image: TestRail offers real-time reporting that helps you meet compliance requirements and keep track of your exploratory tests. TestRail also keeps a transparent chronological history of all notes, screenshots, and defects reported, so you can easily review all your test sessions in a central place. 

The debrief is an often-overlooked information aid in sessions because whoever is acting as a test manager develops a feel for the quality of the application and a holistic view of coverage.  Examples of feedback from a debrief include an evaluation of your progress toward test objectives or a summary of the most important defects detected.

An example of exploratory testing

Here is an example of what exploratory testing might look like in a real-life scenario:

Scenario: You’re a tester working on a web-based e-commerce application for a retail company.

Test charter: Testing the checkout process for usability and functionality.

1. You begin by exploring the application, visiting the website, and understanding its main features, such as product browsing, cart management, and payment processing.
2. To begin your test execution, you add items to your shopping cart and proceed to the checkout. While doing so, you notice that the cart occasionally doesn’t update correctly when you add or remove items. You document this bug with screenshots and a description. 
3. Given this discovery, you adapt and decide to further explore the cart functionality, trying different combinations of adding and removing items and checking for consistency. 
4. You continue documenting bugs you encounter and reporting them to the development team in real time (continuous feedback) until you reach the end of the exploratory session. Bugs could include error messages in the console, unexpected behaviors, and usability concerns.

In this example, this exploratory testing session allowed you to uncover issues with the shopping cart that may not have been identified through scripted testing alone. It also facilitated a quick feedback loop with the development team, contributing to the software’s overall quality.

How to perform exploratory testing using an agile test management tool

Performing exploratory testing using charters is helpful in an agile context because of faster delivery timelines. However, tracking exploratory testing results in a shared spreadsheet can become overwhelming, and details can be lost. Tracking them in individual documents on a network drive can be even worse. 

In TestRail, you can store charters as test cases. You can then create a test run consisting of all the different types of exploratory testing you decide to run for a particular release. 

Image: In TestRail, you can add test cases, choose an “exploratory session” template, add a time estimation, add a mission (purpose of the exploratory session), and add a goal (specific areas of validation). 

The testers can leave comments on the test case in the same place they would for a session, and the software will collect data and make summary reports available on that data.

For simpler applications that combine scripted and exploratory testing, the test run might be two dozen scripted test cases and four charters designed to explore specific risks. The example below records a session as a test case.

Image: Perform exploratory testing using TestRail by recording a session as a test case.

How to report on exploratory testing using a test management tool

Exploratory testing is only effective when you provide stakeholders with accurate and actionable results that provide insights to stakeholders on how to move forward with their processes and products. This includes highlighting key findings, issues, and risks, as well as showcasing coverage, quality, and confidence levels.

Test case management tools can provide an excellent view into exploratory testing. By structuring the sessions between releases as “test cases” within a “test run” (which is the release), any tester or stakeholder with access can see what has been covered and still needs to be covered. 

Image: Use a dedicated test case management platform like TestRail as your exploratory testing tool to manage, organize, track, and streamline the process of producing reports for your exploratory test cases.

Check out this article on how to improve your exploratory testing with a test case management tool!

Exploratory testing FAQs

Scripted testing vs. Exploratory testing:

Scripted testing follows predefined test cases with specific steps, while exploratory testing involves unscripted, ad-hoc testing where testers explore the software intuitively to discover defects and assess its behavior. 

Here’s a table that breaks down the key differences between scripted testing and exploratory testing:

Aspect	Scripted Testing	Exploratory Testing
Test Case Creation	Predefined test cases	No predefined test cases
Test Execution	Follows scripted steps	Ad-hoc, unscripted
Test Planning	Detailed test plans required	Less formal test planning
Flexibility	Less flexible	Highly flexible
Test Documentation	Extensive documentation	Minimal documentation
Test Coverage	Specific and limited	Wider test coverage
Automated Testing	Highly suitable for automated testing	Not suitable for automation (requires human instinct and curiosity)
Bug Discovery	Less effective in finding new issues	Effective in uncovering hidden issues
Use Cases	Suitable for repetitive tasks	Suitable for new feature testing

Types of exploratory testing

Exploratory testing is flexible, unscripted, and can take on various forms based on specific testing goals. Common types of exploratory testing techniques include:

• Freestyle Exploratory Testing: Testers explore the software without specific guidelines or preconceived notions, allowing them to interact with the application freely and uncover defects organically.
• Scenario-Based Exploratory Testing: Testers focus on specific user scenarios or use cases, exploring how the software behaves in different situations. This approach is often used to simulate real-world usage.
• Session-Based Exploratory Testing: Testers organize their exploratory testing efforts into defined timeboxed sessions. During each session, they explore different aspects of the software and document their findings.
• Pair Exploratory Testing: Two testers work together to explore the software. This collaborative approach often leads to discovering defects more effectively as the testers can bounce ideas off each other.
• Ad Hoc Exploratory Testing: This is the most informal type of testing, where testers simply explore the software based on their intuition and experience without any predefined structure or testing strategy.

The type of exploratory testing you choose depends on your testing goals, the software’s nature, and the testing team’s expertise. Often, a combination of testing is used to ensure thorough coverage and defect discovery.

When to use exploratory testing

Exploratory testing is beneficial in the following scenarios and circumstances:

• Early Testing: During the initial phases of a project, when detailed test cases are unavailable, exploratory testing helps uncover issues and validate software functionality.
• Usability Testing: When assessing a product’s user-friendliness and experience, exploratory testing can reveal real user issues that scripted tests may miss.
• Complex or Unfamiliar Systems: When dealing with complex or unfamiliar systems where scripted tests might be incomplete, exploratory testing allows testers to adapt and discover defects organically.
• Rapid Feedback: In agile methodology and iterative development environments, exploratory testing provides rapid feedback to development teams, helping them make quick adjustments.
• Adaptive Testing: When testing needs to adapt to evolving requirements, exploratory testing is more flexible and can respond to changing circumstances effectively.
• Non-Functional Testing: For non-functional attributes like performance, security, and reliability, exploratory testing can simulate real-world usage scenarios to identify vulnerabilities and bottlenecks.
• New Feature Testing: For testing new or recently implemented features, exploratory testing can uncover integration issues and unexpected interactions.

Exploratory testing is valuable when flexibility, adaptability, and the ability to uncover unexpected issues are crucial for effective testing. It complements scripted testing by providing a more holistic view of software quality.

Advantages of exploratory testing in agile

Exploratory testing offers several advantages in agile development environments:

• Flexibility and adaptability: Agile development is characterized by changing requirements and frequent iterations. Exploratory testing’s unscripted nature allows testers to quickly adapt to these changes, making it well-suited for agile teams.
• Rapid and continuous feedback: Exploratory testing provides immediate feedback to Agile teams, allowing them to identify issues early in the development cycle. This helps in swift bug resolution and continuous improvement.
• Meets user needs: Agile development often aims to deliver software that meets real-world user needs and provides value to end-users. Exploratory testing allows testers to simulate actual usage scenarios, uncovering usability and functionality issues that scripted tests might miss.
• Increased creativity: Testers can use their creativity and intuition to discover defects from edge cases that scripted test cases may not cover. They can think like end-users, identifying potential pain points and usability concerns.
• Potential for increased test coverage: Exploratory testing can provide comprehensive test coverage because predefined test scripts do not bind testers. They can explore various paths and interactions, ensuring a broader range of test scenarios.
• Early validation: Agile teams can use exploratory testing to validate user stories and features as soon as they are implemented, reducing the risk of defects accumulating over multiple iterations.
• Risk Mitigation: Agile projects often involve tight timelines. Exploratory testing can help identify high-risk areas early, allowing teams to allocate resources and address critical issues promptly.

Exploratory testing complements scripted testing in agile development by providing the flexibility, adaptability, and quick feedback necessary to keep pace with the agile development lifecycle. It helps agile teams deliver high-quality software that meets evolving requirements and user expectations.

Challenges of exploratory testing in agile

While exploratory testing offers several advantages in agile development, it also comes with some challenges that agile teams should be aware of and address:

• Lack of documentation: Exploratory testing is often less documented than scripted testing, which can lead to less traceability and, ultimately, difficulties reproducing and tracking issues over time.
• Consistency: The unscripted nature of exploratory testing can lead to inconsistency in testing efforts. 

Exploratory testing relies heavily on judgment and intuition. Different exploratory testers may have different perspectives on what constitutes a critical defect or usability issue, leading to subjective assessments.

• Potential coverage concerns: Without predefined test cases, there’s a risk of testers missing important test scenarios or overlooking specific requirements, especially if they are not experienced testers or informed about the project’s scope.
• Scalability: As the project scales, it can become challenging to maintain consistent exploratory testing practices across multiple teams and iterations.

Exploratory testing best practices

While these best practices provide guidance, exploratory testing is a dynamic and flexible approach that you should adapt to suit your project and organization’s specific needs. Here are some general best practices for conducting exploratory testing:

• Define a clear mission: Start with a clear understanding of what you want to achieve through exploratory testing. Define testing objectives and goals to guide your testing efforts.
• Understand the Application: Testers should understand the application’s purpose, features, and user expectations. Familiarity with the domain is also beneficial.
• Use charters: Use test charters or missions to guide your exploratory testing sessions. A charter defines what you intend to explore, helping testers stay focused while allowing flexibility.
• Timebox: Limit the duration of each exploratory testing session with a timebox. This helps maintain focus, prevents overtesting, and ensures efficient use of testing resources.
• Document: Although exploratory testing is less reliant on documentation, it’s essential to document your findings as you go. This includes noting test coverage, risks, test execution, and any questions and issues on the system.
• Focus on risk: Prioritize your testing efforts based on risk. Explore high-risk areas and potential bottlenecks first to uncover critical defects early.
• Feedback Loop: Maintain a feedback loop with the development team. Report defects promptly and engage in discussions to clarify issues and facilitate quicker resolutions.
• Review: After each exploratory testing session, review your findings and reflect on what worked well and what can be improved. Use this feedback to refine your testing approach.
• Test multiple formats: If applicable, explore the software on various platforms, devices, and browsers to identify compatibility issues.
• Reporting: Provide clear and concise reports of your exploratory testing efforts, highlighting significant findings, issues, and recommendations for improvement.