How to Measure Test Automation ROI

For years, managers, practitioners, and agile teams are trying to justify and measure test automation. As achieving sufficient test automation coverage continues to be a major challenge, proving test automation ROI, or the business value of test automation, is more important than ever. 

In the past, teams were trying to measure test automation around the percentage of test cases that are automatable, and the cost savings associated with this task. When putting test automation in a larger business context, measuring test automation ROI should consider the value of the tests themselves and their ability to mitigate and reduce quality risks.

This blog will explore different metrics and best practices for measuring test automation ROI, plus delve into real-life examples of how organizations have benefitted from the proper test automation tooling.  

Table of Contents:

• How Do You Calculate Test Automation ROI?
• Continuous Test Automation & Different Practitioners
• Measuring Test Automation ROI: Key Recommendations and Metrics
• Continuous Testing in DevOps: Metrics
• Demonstrating Test Automation ROI: Business Benefits
   

How Do You Calculate Test Automation ROI?

There is a common formula for ROI measurement:

ROI = Gain – Investment/Investment

The gain will be the objectives and goals you wish to receive from test automation – reduction in feedback, shortened cycles, the release of manual engineers, conversion into revenue, and so on.

Related Reading: ROI of Quality: Making a Business Case for Modern Testing >>

Background: Continuous Test Automation & Different Practitioners

Before covering ways to measure test automation effectiveness, let us review the personas that test automation is serving.

Test automation must be measured not just by automating functional testing by SDETs and/or business testers. Developers within feature teams also take an active part in test automation, hence, such activity needs to consider all practitioners.

Test automation must be measured not just by automating functional testing by SDETs and/or business testers. Other practitioners, such as developers within feature teams also take an active part in test automation and should be able to see its value. 

This means that test automation should have proper metrics that cover different test tools that these personas use daily to achieve their quality as well as their personal goals. If the match between the persona and the tool they use to create test automation scripts is off, the measurements are ultimately irrelevant.

In addition, the usage of the tools by the practitioners needs to also be measured in the context of the test pyramid according to the project’s complexity and needs. Testing the UI or API, functional, integration, aspects of non-functional (load, accessibility, etc.), and unit testing need to be measured accordingly.

When considering the ROI of testing, and test automation specifically, executives and managers need to consider the risks of lack of quality, escaped defects, late releases, etc. rather than just the cost of a software license. This does not imply that there is no need for proper due diligence, considerations, and matching between the test automation and the job to be done.

Lastly, the type of application in many cases will determine a lot of metrics, tools, and considerations teams will take: Web vs. Mobile vs. IoT vs. other or a mix of applications has a great impact on many of the test automation metrics.

Measuring Test Automation ROI: Key Recommendations and Metrics

After establishing the ground for measuring automation by defining personas, application types, test types, and the difference between test automation and test automation in DevOps, let’s focus on measuring and defining solid and high-value test automation.

Since test automation within continuous testing is something teams execute multiple times a day, per each code commit, within CI, and expect value, fast and actionable feedback from – therefore, putting extra emphasis on the creation of the tests is a strong recommendation.

As in the below sketch, three pillars of good test automation include the high value of the tests, high reliability, and fast execution and feedback. If these are part of the pillars, some of the metrics of how to measure test automation ROI must be around these points.

When determining the ROI of test automation the process stages also need to be part of the math:

• Test authoring (time to create a test, ease of use, script reusability).
• Test execution abilities (parallel testing, connectivity, and integration with standard tools and frameworks).
• Test results analysis.
• Test maintenance simplicity.
• Software license cost per user/floating.
• Number of resources required for the above.

Continuous Testing in DevOps: Metrics

For those organizations that are doing continuous testing, here are some other metrics to keep in mind. If your organization is not as mature, these are still good questions to ask yourselves as you look to either start or scale your test automation efforts in the future:

• How quickly are testing activities moving, and what is slowing down these activities?
  • Test flakiness.
  • Test duration.
  • Percentage of automated vs. manual tests.

• Application quality measurements
  • Number of escaped defects and in which areas.
  • MTTD — mean time to detection of the defect.
  • Build quality.

• Pipeline efficiency measurements
  • The number of user stories implemented per iteration.
  • Test automation as part of DoD across iterations.
  • Broken builds with categories.
  • CI length trending.
  • Lab availability and utilization.

• Quality cost measurements
  • Operational costs, lab availability issues.
  • Cost of hardware/software.
  • Costs of defects by severity and stage.

Demonstrating Test Automation ROI: Business Benefits

Another great way to demonstrate test automation ROI and get your top-level management on board is by showcasing actual examples of how organizations have benefitted from investing in the right platform. 

Developer Productivity Gains

Let us take a closer look at three major business cases that managers can make for test automation. The following cases will feature real-life results from prospective Perfecto customers after they improved their testing with the proper tooling.

One important business case to bring up to an executive team is the number of productive hours that developers will gain with the right testing platform.

All developers need to spend time fixing bugs as part of their role. Yet when developers spend too much time on debugging, the company can lose the equivalent of multiple full-time employees.

Not only is this a loss when considering the number of hours resolving issues as part of a developer’s annual salary, but developers are also losing time when they can be coding new features that can bring additional revenue to the company.

With Perfecto, developers can significantly reduce the time spent on fixing bugs and virtually eliminate the need to reproduce defects found in QA. With faster feedback due to testing earlier in the development process, companies have been able to reduce these debugging efforts by as much as 35% and save over $230K annually in developer efficiency gains.

Reduction in Manual Testing Costs

Reducing manual testing costs is another critical point to make to upper management when showcasing test automation ROI.

While manual testing will always play a role in a robust testing strategy, testing manually in areas that can be automated is extremely costly. In one company’s evaluation of their manual testing efforts, they determined that they were losing the equivalent of over 20 full-time employees in manual testing hours every year.

Alternatively, implementing test automation reduces the need for manual efforts significantly while testing a larger breadth of the application in a shorter amount of time. Organizations with a greater level of test automation maturity can also benefit in other areas of testing, such as a higher automation pass rate and increased quality gates.

Customers who have automated 80-95% of their test cases with Perfecto have seen upwards of $1 million in annual savings due to reduced manual testing efforts. While the prospect of that much automation may seem daunting, there is certainly enough ROI to entice upper management to take a closer look at the prospect.

Reduction in Cost Impacts of Escaped Defects

The final business case that will be discussed here covers how modern testing helps reduce the cost of fixing escaped defects. 

According to Perfecto’s customer insights, organizations must contend with a conservative yearly estimate of 120 defects in the delivery pipeline. Each defect is expensive and time-consuming to fix, costing a minimum of $5,000 to support.

By adopting modern testing with a testing platform like Perfecto, organizations can shift left to detect issues early in the testing lifecycle. By catching device-specific issues in pre-production testing, organizations can either significantly reduce or even eliminate escaped defects on the UI/UX level.

Even with a 35% reduction in leaked bugs (which is a considerable percentage), companies can save a minimum of $340K that would otherwise go to fixing bugs later in the development cycle.  

Another important business case for modern testing that deserves an honorable mention is the ability to increase speed to market. While this is something that cannot be quantified as simply as the other business cases, bringing high-quality software to market faster ultimately makes a positive financial impact by attracting and retaining customers. 

Bottom Line

Measuring test automation ROI should focus on more than just license costs and reducing testing cycles. It is a mix of the value, reusability, analysis time, and matching metrics to the project, persona, and technology your teams are using.

The recommendation is to come up with a mix of the above, place some abstract “grades” per each of these items, and prioritize them to understand whether test automation is giving you the value that your team needs.

You can also use these ROI-related calculations to help choose the right tool for your project.

Choosing the right tool requires considering your test development practice (BDD, ATDD, etc.), type of testing (E2E/Visual), and ease of use that includes documentation. Each consideration gets a weight and scoring key that when multiplied by both, will give you a result that can guide your selection.

See Perfecto in Action

Another way to choose the right test automation tool that will give you the most ROI is by trying it out for yourself. Start your free trial with Perfecto today. 

Start Trial

Get Custom Demo

Editor’s Note: The post was originally published in May 2019 and has been updated for accuracy and comprehensiveness

Related Content

• 4 Ways Perfecto Delivers ROI
• Emulation vs. Simulation
• Cloud Testing Is More Critical Than Ever