Manual exploratory testing is an intense activity where you exercise the test application with an objective to explore and understand its behaviors in various situations that are unarticulated explicitly anywhere — be it in the requirements document or user stories. As a result of the exploration, many times, new user flows that were unthought of during the analysis or development phase and bugs in the existing user flows will be discovered. When such discoveries happen, it is refreshingly joyful for the individual who found them as it showcases their complex analytical and keen observation skills!
Typically, manual exploratory testing is carried out in a testing environment, where the entire application is deployed together. The testers take the liberty to meddle with the application components such as the database, services, or the background processes as they please, in order to simulate different real-time scenarios and subsequently observe the application behavior. This exploratory kind of testing differs from manual testing, where the latter refers to the task of manually executing a set of actions described as acceptance criteria in user stories or in the requirements document and verifying if the stated expectations are met successfully. In other words, manual testing doesn’t necessarily exercise any analysis skills. Whereas exploratory testing, as explained before, lays a green field in front of the testers to go above and beyond what is documented and even beyond what is known so far about the application! Given the overlap between manual testing and exploratory testing, some teams, even today, undermine exploratory testing’s value. Also there are beliefs that the amount of analysis carried out as part of user story elaboration and development is good enough to go live, especially when supplemented with automated testing. However, this belief oversees that the analysis carried out during user story creation is majorly from the business point of view, and during development, the developers may focus on the current scope of functionality and may think of test cases around that small piece. Both of these phases leave an obvious gap where the application is not explored with a big picture view and from an end-user perspective in a deployed environment. This gap may leave space for integration issues and missed end-user flows, which is the reason why teams need a manual exploratory testing phase post-development.
Exploratory testing essentially brings all the three angles: the business’ requirements, technical implementation details, and the end-user needs together and certainly challenges everything that is thought of as true from all these angles. Indeed, a good practice is to call a functionality as complete only after the new user flows and test cases discovered from exploratory testing are automated as well.
A point to note here is that the argument is to include a manual exploratory testing phase post-development and not necessarily assign a separate individual to it, although the latter may yield better results due to accumulation of application knowledge and also because it demands someone with keen observation and analysis skills. However, if there are issues such as cost or availability, the team members should take the responsibility of performing exploratory testing in a round-robin fashion every iteration. Indeed, developing exploratory testing skills may even be beneficial for every role to perform better.
If you are one such team member looking to develop your exploratory testing skill, this chapter is for you. We shall discuss the existing frameworks in the industry that can assist in exploratory testing and a strategy to approach exploratory testing in projects. The exercises in the chapter will also enable you to perform exploratory testing on web UI and APIs, specifically. Finally, the chapter throws light on a set of useful practices to maintain test environment hygiene, as a fully deployed test environment plays a vital role in the success of manual exploratory testing.
The exploratory testing frameworks are laid out to essentially help us form mental models that can be intuitively applied to relevant contexts in the application. They try to ease the testing scope by giving clarity and structure to a piece of functionality. To give an example, a common field that exists in almost all applications is a numeric input field. The frameworks lend us structures to logically compart the inputs into sample sets instead of randomly testing all possible numeric values to test the numeric field. Similarly, there are frameworks that try to structure the business rules and thereby help us see the different user flows and test cases
The equivalence class partitioning framework suggests that we split the inputs that result in the same output or undergo similar processing as partition classes and that it is sufficient to pick just one sample input from each partition to test the functionality entirely.
Boundary value analysis extends the equivalence class partition method. The framework calls out explicitly to check for the boundary conditions ineach of the equivalence classes. This specifically helps in finding errors as the boundary conditions are usually vaguely defined and improperly implemented.
The state transition framework assists in deriving test cases, especially when the application’s behavior changes with the history of inputs. For example, the login page would show an error message when the credentials are wrong the first and the second time, but the account may get locked the third time.
The tree, therefore, gives a clear picture of the test cases with their starting states, the action that changes the application’s state, and the expected outcomes to be validated. The visualization also gives us a realistic estimate of the testing efforts for a feature as the number of states and transitions becomes lucid and therefore helps in the planning phase.
There could be more complex state transitions, such as in an order management system where orders go through states like payment complete, shipped, pending, fulfilled, and so on. In such a case, visualizing each state as a node and the actions that take the order to the next state will give a vivid picture of the feature itself to begin with.
When inputs are logically bound (AND, OR, etc.) to produce outcomes, decision tables can be used for deriving test cases. The table especially saves a lot of time during testing as we will have the list of input combinations and expected outputs clearly jotted down ahead. In the login example, the email and password are logically bound by the AND operator, i.e., both email and password have to be right for successful login.
The method also helps in terminating certain combinations, saving us testing time. For example, in the login scenario, the test case 3 where both inputs are incorrect can be eliminated as the login fails even if one of the inputs is wrong.
Cause-effect graphing is another way of visualizing the logically bound inputs and outcomes. The framework mainly helps to view the big picture of a feature and hence finds a prominent space in the analysis phase. Once we have the graph, it can then be translated into a decision table to arrive at detailed test cases.
As seen in the graph, causes and their effects are listed on either side, and the navigation path between them is laid with the help of logical operators.So far, the frameworks showcased their ability to structure the inputs that were related to each other. The next two frameworks will tell us how to deal with multiple independent variables and large datasets.
Many times, we have to deal with more than one input value in applications, and usually, it is a struggle to manage their variations and derive test cases. Pairwise testing or all-pairs testing is a framework that assists in condensing the test cases to a minimum when multiple such independent variables/inputs drive the outcomes. Let’s work on a small exercise to understand the framework.
Consider a form that takes three independent inputs: Operating system(OS) type, device manufacturer, and resolution. The OS field can take two values: Android and Windows; device manufacturers can take three values: Samsung, Google, and Oppo; finally, the resolution field can take Small, Medium, and Large as values. So when we are testing this form.
The new condensed table has cut down the repetition of several pairs. For example, [Google, Medium] and [Google, Windows] pairs occur once now.
So far, we have dealt with inputs/data that are small and consumable by our human brain without the help of tools. But what if we have to test large datasets? For example, let’s say a legacy insurance system has been migrated to a new system, and we have to test if the existing insurance details have progressed correctly into the new system. There could be millions of users in the legacy system, and we cannot apply the frameworks discussed so far to derive test cases. We cannot equally partition the dataset as each user will have their own variations in terms of age, premiums, number of years, scheme type, etc. We cannot apply all-pairs as the variations are too many to identify the recurrence and eliminate them. Sampling is a method that can be applied in such cases. It can be generally applied to any inputs that are continuous and large in nature.
There are two such ‘sampling’ techniques: random sampling and criteria- specific sampling, as seen in Figure. Random sampling is where we pick any data sample from the dataset and verify the results. For example, if there are 1000 users, we can choose 50-100 users randomly from the legacy system and compare it against the new system.
Criteria-specific sampling is where we pick samples by identifying some common characteristics in the dataset. For instance, in the insurance system, we could sample based on user-specific criteria like age, number of years of insurance plan, mode of payment, profession, etc., and insurance policy specific criteria like payment intervals, frequently sold, etc. We could further advance the technique by taking the sample count from each criterion proportional to the actual distribution of the values in the dataset. This would form a representational mini-dataset and likely cover all kinds of test cases.
The last framework we are about to discuss is the error guessing method. It certainly is about exercising your analytical and logical thinking more than providing guidelines. Let’s look into it now.
Error guessing is the method where software professionals predict a failure based on their past experiences. The predictions are common misses in integration, input validations, boundary cases, and more. Even though past experiences play a vital role in predicting probable error cases, you could also use your understanding of the technology and logical reasoning to predict such errors. Promoting this kind of thinking indeed boosts your exploratory testing skill in large proportions.
Here are a few common errors that occur recurrently in my observation:
The validations for invalid/blank input values will be missed. There should be an appropriate business error message directing the user to correct the input.
The HTTP status codes returned for data validations, technical and business errors are usually unclear. HTTP status codes are discussed in detail later in this chapter as well.
Boundary conditions specific to the domain, data types, states, etc., are commonly left unhandled.
Technical errors such as server down, response timeouts, etc., are left unhandled on the UI side.
The UI has jerks and residues during transitions, data refreshes, and navigations.
The SQL keywords ‘like’ and ‘equals’ will be interchangeably used, changing the results entirely.
Caches will be left uncleared, and session timeouts will be undefined.
When clicking the browser back, the request may get posted again.
Missing file format validations when uploading files from different OS platforms.
Suppose you are asked to perform exploratory testing on the order creation functionality of an eCommerce application today. What are the discovery paths you will kickstart with? This section answers that question by throwing light on four essential paths that need to be mandatorily explored in any given application.
Functional user flows of an application refer to the journey an end-user takes on the application, such as logging in, searching for a product, adding it to the shopping cart, providing the shipping address, paying for the order, and finally, getting an order confirmation. This is a single positive user flow, which is the first and foremost to be validated. You will have to explore the positive flow with different shipping addresses, delivery methods, and item combinations to ensure it works entirely.
While exploring, you may find yourself needing the pack of eight exploratory testing frameworks discussed earlier. For example, you may need the equivalence class partitioning and boundary value analysis methods to verify if the application adds the right tax amount to the total price. Similarly, you can use the transition tree to derive test cases around the shipping address and delivery methods available to that address combination. Once you ensure the single user positive flow works perfectly, you should start exploring the repeat flow and multiple users flow next:
Typically, end-users repeat their flows multiple times on the application, like searching for products and adding them to the shopping cart repeatedly. But commonly a user flow is tested once, and if it works, the repeat flows are ignored under the assumption that the behavior will remain the same. But practically, this assumption may or may not be true. For example, when the user tries to add the same item again to the shopping cart, the UI may show a message saying the product is already added and check if they want to increase the quantity.
A functionality may work perfectly from a single user’s point of view, but that may not be sufficient as several users consume the application simultaneously in real-time. So exploring the collision scenarios, where one user’s actions impact the other user, becomes important. For instance, what happens when two different users add the last available product to their shopping carts at the same instance?
To summarize, functional user flows can be chosen as the first discovery path to explore in an application, and within that path, there could be several sub-branches to explore. A few essential sub-branches are single users’ positive flows, repeat flows, and multiple users’ flows.
As called out at the beginning of the chapter, exploratory testing is carried out in a test environment where the testers meddle with the application components to simulate real-time scenarios and observe the application behavior. There are two clauses in that statement, which need your attention: meddling with the application and real-time scenarios, as they form the core of exploratory testing. When thinking of real-time scenarios, you should also think of all possible failures, as failures are practically unavoidable. For instance, there could be a network failure between the application components, and hence they may not be able to respond back to the user. Or the network could be slow between the end-user and the application server, resulting in delays for the end-user. Or the application services could be down due to hardware failures. All of these failures have to be anticipated during exploratory testing and simulated in the test environment.
Other than the aforementioned failures (network, service, and hardware failures), there could be errors due to invalid user actions. A functionality can be deemed complete only if it has built-in validations to handle such invalid user actions indeed. In the order creation functionality, several instances call for exploring the validations. For example, as discussed in the frameworks section, the login page needs to have validations on the email and password; the search text entered for searching a product needs to be validated for invalid inputs, availability of the item, and so on. The other places are the shipping address and payment details, adding items to the shopping cart, etc. Exploratory testing should give a major stake to such failures and error handling. As a result of such error handling, you should observe if the functionality advises the users on their mistakes, if any, and suggests remediations through meaningful error text.
The UI is what the end-user sees, and there can’t be obvious lags in the quality of the UI. So, this discovery path is one of the major ones to explore. A few UI quality related test cases in the order creation functionality are: the eCommerce UI should provide adequate space for different lengths of shipping addresses. It can’t be congested when the length is too long, and it can’t be spread out too much with empty spaces when the address is too short. Similarly, the product images have to be shown with the right quality; end-users should be able to seamlessly operate the application from their preferred browsers, and there should be a loading icon when there are delays.
There are several cross-functional aspects in any given functionality such as security, performance, accessibility, authentication, authorization, auditability, privacy, and so on, that require specific focus during exploratory testing. Many of these aspects have taken a chapter for themselves in this book because of their significance. To give a brief on some of these aspects, especially from the point of view of exploratory testing the order creation functionality:
In the order creation user flow, an abusive user could enter SQL queries in the UI input fields and try to hack the application. The application should have validations to handle them. Similarly, the credit card details should not be stored in plain text in the application database and shouldnot be logged in plain text in the application logs. This is to secure them even in the case of a potential breach.
User’s private data such as credit card details and shipping address should not be stored in the application database without getting consent from the user. Also, the way in which their data might be used for analytics or sent to third-party services for processing should be informed to them in advance. Several data privacy clauses are also enforced by legal regulations
Most websites have user authentication functionality, which would call for exploring authentication-related test cases such as single sign-on, two-factor authentication, session expiry, account locking, unlocking, etc. In the eCommerce application, end-users may be allowed to view the product catalog without login but may not be allowed to place an order without login.
Similarly, there could be roles (e.g., admin, customer executive) and permissions (e.g., editing an order) assigned to each user, which requires exploring the authorization-related test cases such as multiple overriding roles, new permissions getting added to existing roles, observing the application behavior when an operation is executed without right permissions, etc.
The manual exploratory testing strategy, depicted in Figure, ties all that we discussed and bundles the team processes, giving a practical direction to apply exploratory testing in day-to-day project work. Let’s start from the outer semicircle and move inwards.
The first semicircle emphasizes understanding the application details in totality and points to five broad application areas that you should particularly collect details on. Learning those details will essentially help you kickstart exploratory testing, but as mentioned earlier, you will certainly find new information about the application during the exploration.
Here is a brief on the five broad areas that you may focus on while trying to understand the application:
A persona is a character that represents the demography of end-users with similar attributes. In software teams, such user personas are created at the beginning of the project itself so that their specific needs can be imbued into all the stages of the delivery lifecycle, starting from design. An example of user personas impacting the features of an application is the social networking site, where young adults may expect an extravagant experience and seniors may expect a clean and clear interaction. And when testing is all about wearing the end-user’s hat, knowing the set of user personas the application intends to serve and exploring how each persona will perceive and interact with the application is vital.
Every domain such as social networking, transport, health, etc., has a tailored workflow, process, and terminology/jargon that needs to be understood to kickstart exploration. ECommerce is another perfect example where domain knowledge becomes critical in testing. For instance, an order once created goes through a defined workflow such as capture, promise, confirm, and so on. The order fulfillment flow has to interact with so many parties such as the warehouse which has the items, the shipping partner which ships the items from the warehouse to the customer, and the vendors to replenish the items regularly. So while observing the application behavior during exploratory testing, you need to know when to flag the observation as a deviation and you may find it hard to do that without the domain knowledge.
Consider the scenario where the business priority is to design the solution as a platform for extendability and scalability purposes. In such cases, just testing the functional user flow from the UI may not be sufficient. It needs to be explored from a ‘platform point of view’ and observed if the UI and web services are not tightly coupled and if the web services are independent to be integrated with other systems and other similar angles.
Knowing the application architecture will add branches to your discovery paths in an exploratory testing session. For example, if an architecture involves web services, you may also need to perform API exploratory testing (discussed later in this chapter in detail) apart from just exploring the UI. Similarly, if the application involves event streams, exploring the cases around asynchronous communication becomes important. So understanding the architecture on a high level will help carve the discovery pathways in terms of internal component integrations, data flow between components, third-party integrations, and error handling. Several of these aspects are discussed throughout the book as well.
As discussed earlier, meddling with the test environment to simulate real-time scenarios, including failure cases, is the means of performing exploratory testing. To do the same, information about which application components are deployed where and the configurable levers will give critical hints to finding new discovery paths. For example, web services may be configured with the maximum number of hits they can serve within a time period (also referred to as rate-limiting). 2 You may then need to observe the application behavior when the rate limit exceeds.
So, gather some basic information around infrastructure and configurations such as how the services and database are deployed—in a single machine or spread across multiple machines, rate limit settings, API gateway settings, etc.
Once you have gathered enough information around these five application areas, you are set to delve into the actual exploratory testing phase.
In his 2003 paper, ‘Exploratory Testing Explained,’ James Bach says, “Exploratory testing is simultaneous learning, test design, and test execution.” It is one of the widely used definitions of exploratory testing till now. To elaborate what he said, exploratory testing is performing a chain of actions on the application, observing the behavior, and thereby learning more about the application and exploring it incrementally. Such a process shall demand our brains to be alert all the time and do in-depth analysis. We as humans can pay intense attention and get to the depths only by focusing on smaller scopes of work. This is why we should explore individual parts of the application at a time! The individual part could be any of the previously discussed discovery paths or a sub-branch in the path, such as a user flow, a feature, or a cross-functional aspect like security.
It may be overwhelming to manage all these paths and sub-branches while exploring in-depth parallelly. So, one way to keep track of all of the individual parts and the sub-branches you plan to explore is to use a mindmap
This style of exploring in parts is represented as the second inner semicircle of the strategy diagram in Figure. As we saw earlier, we may need the help of the pack of eight testing frameworks while exploring, which is represented as the third inner semicircle.
Exploratory testing cannot be a one-time activity. Every day the team continues to add new code, new features, and new integrations, resulting in changes to the application behavior and thereby calling for exploration again. So considering exploratory testing as a continuous process will allow you to structure the scope that can be explored at depth in a particular time or phase. For example, some agile teams have the practice of doing dev-box testing. During dev-box testing, the business representatives and the testers perform time-bounded exploratory testing of the user story that just got developed in the developer’s machine itself. Here you can restrict the scope to only the positive user flows, validations and UI look and feel. The next phase conducive for exploration is the user story testing phase. Here you can expand the scope for exploration to include some of the cross-browser and cross-functional aspects.
Similarly, some agile teams conduct bug bashes frequently, where all team members sit together and explore the application features developed so far. And finally, at the release testing phase, you can focus on the cross- functional aspects in-depth such as performance, reliability, scalability, etc., and explore the positive user flows and integrations at a slightly higher level. So plan your exploratory testing phases along with its scope ahead. This will help the team get continuous feedback and, therefore, allow space for continuous improvement.
To consolidate the strategy, whenever you are starting exploratory testing, first, understand the application details, jot down the individual paths to explore, and continue the exploration of different pathways in different phases of the delivery cycle in order to provide continuous feedback to the team.
The test environment is the actual playground where the testers apply their exploratory testing skills, and when it is poorly maintained, it directly restrains the testers and their outcomes. We will discuss a few such maintenance smells, their impact, and remedies to overcome them.
In several large teams, a single test environment is shared among the sub-teams, and this places heavy restrictions on the individual teams’ testers from meddling with the environment as per their discovery paths. For example, if they need to bring a service down momentarily, they need to get consent from other teams. Worse is that they need to coordinate with other teams or wait until the regular deployment schedules, which maybe once a day or a week, to take a new deployment and explore the latest code. Thus, having a dedicated test environment, at least with the components that come within the boundaries of the individual sub-team, will give their testers the liberty to explore adventurously.
Once the team has their own dedicated environment, a suitable approach is to have a manual trigger in the Continuous Integration (CI) pipeline to take new deployments instead of automated deployments as it could alter the tester’s configurations on the environment without warnings. Also, the build should be made available for deployment in the CI only when the automated test run stage has passed. This is to ensure that there are no open defects in the latest code, which can block exploratory testing.
Also, the test environment should be set up as close to the production environment with firewalls, separated tiers/components, rate limit configurations, etc., as part of the deployment. Only then can the failure test cases discussed earlier can be explored.
Test data comes under the purview of the testers, and they should follow certain practices to ensure that they don’t unintentionally derail their own exploration. A few callouts on test data hygiene are: be wary of stale data and configurations when continuing to test a new feature in the same deployment. A recommendation to avoid such complications is to make it a point to deploy a new build whenever starting a new user story. This is assuming that a new deployment would clean the old data and configurations and revive the application back to a fresh state. Alternatively, make it a point to create a new set of test data for every user story, such as a new user instead of exploring with existing users, which could be in different states.
Sometimes, test data creation is not that simple at all when there are hundreds of linked tables. An option then is to have the new deployment delete the old data, dump a standard set of test data or have a SQL script to create new test data as part of it. Another option is to use the production data after anonymizing the private details in the testing environment, although it may warrant concerns from a security perspective if not careful.
Most often, the environment-related accesses are not available to team members. They will have to work with the devOps engineers outside the team to even look at application logs or setup stubs. This especially gets troublesome during exploratory testing, where the testers may need to access all the application components. So ensuring the team is autonomous with required access will cut down delays due to external dependency.
Usually, the third-party services are left out of the test environment setup, assuming the integration can be tested directly in production. This may result in unwanted blockers, especially when it is too late in the delivery cycle. Hence, while thinking of test environment setup, a way to explore the integration with third-party services either by employing stubs or by buying limited access to the third-party services should be considered early.