We have introduced a new feature: Test Execution with Multiple Datasets. In this blog, We’ll explore how to create tests encompassing multiple datasets, execute them, and leverage the insights provided by the reports and the analysis graph.
Understanding the Feature
Prior to this update, useMango supported test execution with a single dataset. Now, we’ve introduced the capability to execute tests using multiple datasets. This means you can drive test behaviors using variant data values, eliminating the need to execute each datasets one at a time.
To better understand this feature, consider you’re testing an e-commerce website’s shopping cart functionality by searching and adding an item to the cart based on the name and the category. Each item variant represents a dataset containing the item’s name and category. Before this feature was available, useMango allowed executing these datasets individually, necessitating manual selection for each run, demanding more effort and maintenance. However, with this newly introduced feature, it’s now possible to select multiple datasets at once which will then execute each dataset sequentially and a consolidated report will be generated encompassing all dataset details.
In the upcoming sections, we will use the above scenario to create a test with multiple datasets, execute that test and view the consolidated report that is generated for the test execution
Note
Please take note that what were previously referred to as ‘Scenarios’ in useMango are now identified as ‘Datasets’.
Creating Tests with Multiple Datasets
Let’s consider the scenario of testing an e-commerce website’s shopping cart functionality across different product categories.
For the sake of brevity, a simple test has already been created to demonstrate execution with multiple datasets.
This test validates useMango practice website’s functionality of adding a product to the cart, given the product category and the name as a parameter using the multiple datasets feature. It launches the website, searches for the product, adds it to the cart, verifies its presence in the cart, and finally, closing the browser window.
To learn how to create tests using multiple datasets in useMango, check out this guide Explore how to create tests with multiple datasets in useMango.
Attached below are the screenshots of the test and the datasets used.
Now the test is ready to execute! Let’s execute it with multiple datasets.
Executing Tests with Multiple Datasets
Running tests with multiple datasets in useMango is a seamless process. Once your test is configured to handle different scenarios through datasets, executing these tests becomes a matter of a few clicks.
To execute click on the test and hit the run button. This will open a popup that lets you select the datasets for the execution as shown below.
To know more refer to our documentation for a step-by-step guide on executing tests with multiple datasets.
After clicking on the “Run Test” button the test executes sequentially for every dataset selected.
Upon executing the test, the control returns to the useMango app. For the selected test, under recent runs, simply click on the right most icon—a stack of disks symbolizing multiple datasets executions differentiating it from the single dataset execution represented by the encircled icon, clicking on this icon opens up the consolidated report in the app website which we will look at in detail in the next section.
Multiple Datasets Reporting
The consolidated report that is generated after the execution provides a summarised result distribution chart along with execution details of each dataset. This report offers a comprehensive overview of the test’s performance across different datasets.
The following presents the consolidated report for the same execution:
The report contains a pie chart displaying the Passed/Failed/Cancelled/Not Run result distribution of the datasets executed. It also displays other execution details like Started At time, Total Duration, User email and so on. The consolidated summary report PDF can be downloaded by clicking on the download button besides the execution result at the top right.
Additionally, a table displays individual dataset execution details. By clicking on the dataset name, you can view the specific report for that dataset’s execution. For each dataset, both a summary and a detailed report are available to download via their respective download icons.
Consolidated Summary Report PDF
The summary report PDF contains the result distribution pie chart and information of the datasets and a step wise summary of each dataset executed. It offers the advantage of easy sharing via email or as an attachment, facilitating seamless collaboration and comprehensive test result discussions among team members.
Executions Page
You can view all the executions of the project from the executions page within the app website. This list is sorted by the most recent execution, so we see the test we executed is at the top of the list in the image below.
This execution has a filled stacked disc icon at the beginning of the row, to represent the execution run with multiple datasets and differentiate it from the other executions.
Clicking on an execution’s name opens the consolidated report, if it is an execution run with multiple dataset. Otherwise it opens up the single dataset report.
We will delve into Execution Analysis in the next section.
Execution Analysis
In this section, we’ll go over the historical analysis of executions and how executing multiple datasets impacts the analysis graphs.
Within the “Reports” tab on the app website, you can visually analyze your project’s execution progress over time through two available reports: Execution History and Execution Time.
Execution History
Access the execution history report by navigating to the “Reports” tab. This shows you the execution count report which displays the count of the executions over the specified time period.
In this report, we can’t display executions involving multiple datasets and single test executions together as they don’t mean the same thing. Therefore, an “Execution Type” filter is provided. By default “Single Dataset” filter is set that displays single dataset executions i.e. execution with no datasets, single dataset and also executions that were part of an execution containing multiple datasets.
If you wish to see the executions with multiple datasets, select “Multiple Datasets” from the Execution Type filter.
Clicking on the bar in the report directs you to the executions’ list which displays the executions based on the filters that were applied.
Below is the executions list when the Execution Type filter is set as “Single Dataset” on the reports page. Executions listed without an icon denote standalone runs on single datasets, while those marked with partially filled stacked disc icons represent single dataset executions which were part of multiple datasets execution.
Below is the executions list when the Execution Type filter is set as “Multiple Datasets”. The executions that represent multiple datasets runs are represented by a filled stack disc icon.
Execution Time
To view the Execution Time report select the “Execution time” from the select report dropdown on the “Reports” tab.
This report will only work on single dataset executions, encompassing execution with no datasets, single dataset and executions that were part of a multi-dataset execution. However, we do not plot Execution time at the level of multi-dataset execution.
To delve into useMango’s automation capabilities, including its “Test Execution with Multiple Datasets” feature, begin your free trial now and revolutionise your testing process!