How to Calculate the Value of Analytics Solutions

With the move to DevOps, automated testing is shifting to continuous testing.

Continuous testing (CT) produces a high volume of data. The amount of information generated by a single project is substantial and requires a methodological approach. Because of this, continuous testing mandates the transition into full-blown test analytics.

It’s critical for DevOps teams to implement an analytics solution in order to derive meaning from their test results. Failure to do so dissolves the benefits of CT and even DevOps as a whole.

On the other hand, a solid analytics solution can provide a number of benefits, including:

• Increased visibility into emerging testing issues.
• Reduced noise.
• Improved response time for triaging issues.
• Expanded high-level visibility as testing scales.

As with any business decision, you need to carefully weigh your options. Keep reading to learn more about how analytics solutions can help testing and dev teams, and learn how to determine the value of an analytics solution yourself.

Understanding the Value of Analytics Solutions

A good analytics solution can give you fast feedback and visibility, which saves you time and a lot of money.

In fact, DZone estimates that the average developer can spend 80% of their time replicating errors and fixing them. That’s too much time that should be spent on product development rather than on bugs.

The median annual salary of a developer is $105,590 across the US. That means $1,600 a week is spent on each developer fixing errors. If you could reduce that by half for each of your developers, you’d save about $40,000 a year.

You can also calculate the cost yourself:

1. Determine the number of tests you execute in a typical project.
2. Multiply that number by your test failure rate. This provides you with the number of test executions you need to review.
3. Multiply the number of test executions you need to review by the time it takes to review each failure.
4. Multiply the total time of review by the cost of labor.
5. Compare the total cost with the cost of an analytics solution.

This gives you an idea of what it costs to do it yourself. Compare this bare-bones DIY process to full-blown analytics solutions. You’ll soon discover that you may not be saving as much as you thought. This is especially troubling when you consider that amount will only continue to grow as your tests scale.

How Many Tests Do You Need to Run?

For the sake of calculation, let’s assume that you have an average of 500 test cases per typical project, running on an average seven platforms on a monthly basis. If running once every four weeks, this means:

500 test cases X 7 platforms = 3,500 test executions.

In this continuous testing scenario, you would likely need to increase to two vectors. For the number of test cases, let’s assume an increase of 50% to 750 test cases and an increase in frequency from once every few weeks to daily.

On a daily basis, you will need to execute the following:

How Many Test Executions Do You Need to Review?

Previously, there was one (or very few) decision points in “post cycle” testing scenarios. This means you would need to at least review the failures of the test execution once a month. With continuous testing, you do the same, but on a daily basis. And, with increased executions.

The failure rate (of both true and false negatives) varies according to organizations, skills sets, environments, and other factors related to the project. But 25% is considered to be on the good side. And, a typical review takes 2.5 minutes per execution.

So, consider this post cycle scenario:

Five days of reviewing once a month is not ideal. But, it’s still in the “acceptable” range.

Now, consider this in-cycle scenario:

That’s an impossible feat if you’re supposed to review these once a day. And, we’re not even getting into any advanced analysis. This is just analysis to determine what happened.

As you can see, this approach is not suitable for continuous testing and the requirements of modern DevOps. Instead, testing teams need test analytics, which allow you to find the cause of the failures and get granular information on how to fix them.

How to Get Meaningful Analysis: Cause of Failure

When you test continuously, you end up with a mountain of data. To allow analytics solutions to give meaningful results, it needs to include failure reasons for each test execution. This can be done in three ways.

1. By a Human — Inefficient and extremely expensive.
2. Post Analysis — Take all results. Review them one by one. Figure out what just happened. This approach is fair, but limited, since some information no longer exists (app crash reports, network data, etc.).
3. Real-Time Automatic Analysis — This is similar to Post Analysis. But, during the execution, more data is collected and categorized (like an app crashing). This approach yields the best results. However, keep in mind that real-time automatic analysis is only possible if provided by the execution platform, such as Perfecto.

1. By a Human

This method is not very efficient and it’s extremely expensive. At $38/hour, 55 hours of review costs $2,090 per daily cycle or $460,000 annual spending. And, that’s without management cost, hardware and software, and other upkeep expenses.

Keep in mind that this option gives you an entire day of delays in analytics — which is not the fast feedback DevOps requires.

2. Post Analysis

For this option, you need an analytics solution, such as Splunk or Tablau. You will also need a data scientist to analyze the data, find patterns, and maintain it over time.

A Splunk-based option will cost about $266,000 a year, including the price of a data engineer. A non-Splunk-based option will cost $158K/per year per project.

Note that the level of information achieved by this option is based off the level of information provided by the test execution platform. If information is provided (even in raw format), it can be extracted. If not, it will still require manual labor.

In some cases, users will be able to identify defects during script executions and if given the ability to mark it in the results (with tags, custom failure reasons, etc.) It can be used for this purpose as well. 

📕 Related Resource: Learn more about How to Use Bolt Tasks for Better Splunk Compliance Reporting

3. Real-Time Automatic Analysis

This option is only possible if provided by the execution platform — such as Perfecto, which offers the only real-time analytics solution on the market.

Perfecto processes each test during and after its execution. It categorizes the execution failures into dozens of possible reasons. These possible reasons can stem from problems with:

• The scripts
• The backend
• The network
• The application itself

The Perfecto “failure reasons” functionality also allows you to customize this list with your own unique categories.

Once information is available, Perfecto can slice-and-dice the information and present the user with quick feedback about what happened, why it happened, and what they need to do about it. This saves developers’ time and leads to a quick resolution.

Perfecto’s analytics also offers the following:

• Overview of testing throughout the CI pipeline in real-time and with trends.
• Heatmap dashboard of emerging issues in testing.
• Cross-platform visual validation of functional/UI defects.
• AI-based noise reduction through detection and classification of failures.
• Single Test Reports for detailed root-cause-analysis with a rich set of artifacts.
• Report Library list for effective triaging.
• "Single pane of glass" provides visibility and scales to support millions of test results.

Analytics Solutions Make Continuous Testing Better

As you can see, analytics can help teams test smarter and faster. There are many factors to consider when deciding between analytics solutions.

You need to ensure that your solution scales as you grow. It needs to be smart and intuitive, giving insights into test failures. And, it should provide you with the visibility you need to quickly resolve any test failures you encounter.

Perfecto’s smart test automation platform for web and mobile testing includes smart analytics. And it’s the only platform to truly offer real-time automation analysis. Not only that, but it’s also the only end-to-end solution for all your testing needs — from test creation to execution and analysis.

To learn more, check out check out Perfecto’s Reporting and Analytics to see how to test smarter and work faster.

Smart Reporting and Analytics

Related Content

• Top 5 Features Your Test Analytics Tool Needs Right Now
• Top 5 Challenges in Automation Testing and Analytics (and How to Overcome Them)
• 5 Reasons Why You Shouldn't Stitch Together Testing in the DevOps Pipeline