How to Measure Test Automation ROI

Measuring test automation efficiency, its ROI (return on investment), and value to the overall quality of a product is an especially important discussion in the world of test automation. For years, managers, practitioners, and agile teams are trying to justify and measure test automation. As achieving sufficient test automation coverage continues to be a major challenge, proving the value of test automation is more important than ever. 

In the past, teams were trying to measure test automation around the percentage of test cases that are automatable, and the cost savings associated with this task. When putting test automation in a larger business context, measuring test automation ROI should consider the value of the tests themselves and their ability to mitigate and reduce quality risks.

Continuous Test Automation & Different Practitioners

Before covering ways to measure test automation effectiveness, let us review the personas that test automation is serving.

Test automation must be measured not just by automating functional testing by SDETs and/or business testers. Developers within feature teams also take an active part in test automation, hence, such activity needs to consider all practitioners.

Test automation must be measured not just by automating functional testing by SDETs and/or business testers. Other practitioners, such as developers within feature teams also take an active part in test automation and should be able to see its value. 

This means that test automation should have proper metrics that cover different test tools that these personas use daily to achieve their quality as well as their personal goals. If the match between the persona and the tool they use to create test automation scripts is off, the measurements are ultimately irrelevant.

In addition, the usage of the tools by the practitioners needs to also be measured in the context of the test pyramid according to the project’s complexity and needs. Testing the UI or API, functional, integration, aspects of non-functional (load, accessibility, etc.), and unit testing need to be measured accordingly.

When considering the ROI of testing, and test automation specifically, executives and managers need to consider the risks of lack of quality, escaped defects, late releases, etc. rather than just the cost of a software license. This does not imply that there is no need for proper due diligence, considerations, and matching between the test automation and the job to be done.

Lastly, the type of application in many cases will determine a lot of metrics, tools, and considerations teams will take: Web vs. Mobile vs. IoT vs. other or a mix of applications has a great impact on many of the test automation metrics.

Measuring Test Automation ROI: Key Recommendations and Metrics

After establishing the ground for measuring automation by defining personas, application type, test types and the difference between test automation and test automation in DevOps, let’s focus on measuring and defining solid and high-value test automation.

Since test automation within continuous testing is something teams execute multiple times a day, per each code commit, within CI, and expect value, fast and actionable feedback from – therefore, putting extra emphasis on the creation of the tests is a strong recommendation.

As in the below sketch, three pillars of good test automation include the high value of the tests, high reliability, and fast execution and feedback. If these are part of the pillars, some of the metrics of how to measure test automation ROI must be around these points.

When determining the ROI of test automation the process stages also need to be part of the math:

• Test authoring (time to create a test, ease of use, script reusability).
• Test execution abilities (parallel testing, connectivity, and integration with standard tools and frameworks).
• Test results analysis.
• Test maintenance simplicity.
• Software license cost per user/floating.
• Number of resources required for the above.

Continuous Testing in DevOps: Metrics

For those organizations that are doing continuous testing, here are some other metrics to keep in mind. If your organization is not as mature, these are still good questions to ask yourselves as you look to either start or scale your test automation efforts in the future:

• How quickly are testing activities moving, and what is slowing down these activities?
  • Test flakiness.
  • Test duration.
  • Percentage of automated vs. manual tests.

• Application quality measurements
  • Number of escaped defects and in which areas.
  • MTTD — mean time to detection of the defect.
  • Build quality.

• Pipeline efficiency measurements
  • The number of user stories implemented per iteration.
  • Test automation as part of DoD across iterations.
  • Broken builds with categories.
  • CI length trending.
  • Lab availability and utilization.

• Quality cost measurements
  • Operational costs, lab availability issues.
  • Cost of hardware/software.
  • Costs of defects by severity and stage.

Finally, as an important reminder, there is the common formula of ROI measurement.

ROI = Gain – Investment/Investment

If to use the above formula and metrics, while also referring to an old but still relevant blog from TestingWhiz, we can see the following: The gain will be the objectives and goals you wish to receive from test automation – reduction in feedback, shortened cycles, release of manual engineers, conversion into revenue, and so on.

Bottom Line

Measuring test automation ROI should focus on more than just license costs and reducing testing cycles. It is a mix of the value, reusability, analysis time, and matching metrics to the project, persona, and the technology that your teams are using.

The recommendation is to come up with a mix of the above, place some abstract “grades” per each of these items, and prioritize them to understand whether test automation is giving you the value that your team needs.

You can also use these ROI-related calculations to help choose the right tool for your project.

Choosing the right tool requires considering your test development practice (BDD, ATDD, etc.), type of testing (E2E/Visual), and the ease of use that includes documentation. Each consideration gets a weight and scoring key that when multiplying both, will give you a result that can guide your selection.

Editor’s Note: The post was originally published in May 2019 and has been updated for accuracy and comprehensiveness

Related Content

• 4 Ways Perfecto Delivers ROI
• Emulation vs. Simulation
• Cloud Testing Is More Critical Than Ever