View all web browser and mobile devices available in our cloud-based test lab.
With the move to DevOps, automated testing is shifting to continuous testing.
Continuous testing (CT) produces a high volume of data. The amount of information generated by a single project is substantial and requires a methodological approach. Because of this, continuous testing mandates the transition into full-blown test analytics.
It’s critical for DevOps teams to implement an analytics solution in order to derive meaning from their test results. Failure to do so dissolves the benefits of CT and even DevOps as a whole.
On the other hand, a solid analytics solution can provide a number of benefits, including:
As with any business decision, you need to carefully weigh your options. Keep reading to learn more about how analytics solutions can help testing and dev teams, and learn how to determine the value of an analytics solution yourself.
Table of ContentsUnderstanding the Value of Analytics SolutionsHow to Get Meaningful Analysis: Cause of FailureAnalytics Solutions Make Continuous Testing BetterRelated ContentTable of Contents1 - Understanding the Value of Analytics Solutions2 - How to Get Meaningful Analysis: Cause of Failure3 - Analytics Solutions Make Continuous Testing Better4 - Related Content
A good analytics solution can give you fast feedback and visibility, which saves you time and a lot of money.
In fact, DZone estimates that the average developer can spend 80% of their time replicating errors and fixing them. That’s too much time that should be spent on product development rather than on bugs.
The median annual salary of a developer is $105,590 across the US. That means $1,600 a week is spent on each developer fixing errors. If you could reduce that by half for each of your developers, you’d save about $40,000 a year.
You can also calculate the cost yourself:
This gives you an idea of what it costs to do it yourself. Compare this bare-bones DIY process to full-blown analytics solutions. You’ll soon discover that you may not be saving as much as you thought. This is especially troubling when you consider that amount will only continue to grow as your tests scale.
For the sake of calculation, let’s assume that you have an average of 500 test cases per typical project, running on an average seven platforms on a monthly basis. If running once every four weeks, this means:
500 test cases X 7 platforms = 3,500 test executions.
In this continuous testing scenario, you would likely need to increase to two vectors. For the number of test cases, let’s assume an increase of 50% to 750 test cases and an increase in frequency from once every few weeks to daily.
On a daily basis, you will need to execute the following:
Previously, there was one (or very few) decision points in “post cycle” testing scenarios. This means you would need to at least review the failures of the test execution once a month. With continuous testing, you do the same, but on a daily basis. And, with increased executions.
The failure rate (of both true and false negatives) varies according to organizations, skills sets, environments, and other factors related to the project. But 25% is considered to be on the good side. And, a typical review takes 2.5 minutes per execution.
So, consider this post cycle scenario:
Five days of reviewing once a month is not ideal. But, it’s still in the “acceptable” range.
Now, consider this in-cycle scenario:
That’s an impossible feat if you’re supposed to review these once a day. And, we’re not even getting into any advanced analysis. This is just analysis to determine what happened.
As you can see, this approach is not suitable for continuous testing and the requirements of modern DevOps. Instead, testing teams need test analytics, which allow you to find the cause of the failures and get granular information on how to fix them.
When you test continuously, you end up with a mountain of data. To allow analytics solutions to give meaningful results, it needs to include failure reasons for each test execution. This can be done in three ways.
This method is not very efficient and it’s extremely expensive. At $38/hour, 55 hours of review costs $2,090 per daily cycle or $460,000 annual spending. And, that’s without management cost, hardware and software, and other upkeep expenses.
Keep in mind that this option gives you an entire day of delays in analytics — which is not the fast feedback DevOps requires.
For this option, you need an analytics solution, such as Splunk or Tablau. You will also need a data scientist to analyze the data, find patterns, and maintain it over time.
A Splunk-based option will cost about $266,000 a year, including the price of a data engineer. A non-Splunk-based option will cost $158K/per year per project.
Note that the level of information achieved by this option is based off the level of information provided by the test execution platform. If information is provided (even in raw format), it can be extracted. If not, it will still require manual labor.
In some cases, users will be able to identify defects during script executions and if given the ability to mark it in the results (with tags, custom failure reasons, etc.) It can be used for this purpose as well.
📕 Related Resource: Learn more about How to Use Bolt Tasks for Better Splunk Compliance Reporting
This option is only possible if provided by the execution platform — such as Perfecto, which offers the only real-time analytics solution on the market.
Perfecto processes each test during and after its execution. It categorizes the execution failures into dozens of possible reasons. These possible reasons can stem from problems with:
The Perfecto “failure reasons” functionality also allows you to customize this list with your own unique categories.
Once information is available, Perfecto can slice-and-dice the information and present the user with quick feedback about what happened, why it happened, and what they need to do about it. This saves developers’ time and leads to a quick resolution.
Perfecto’s analytics also offers the following:
As you can see, analytics can help teams test smarter and faster. There are many factors to consider when deciding between analytics solutions.
You need to ensure that your solution scales as you grow. It needs to be smart and intuitive, giving insights into test failures. And, it should provide you with the visibility you need to quickly resolve any test failures you encounter.
Perfecto’s smart test automation platform for web and mobile testing includes smart analytics. And it’s the only platform to truly offer real-time automation analysis. Not only that, but it’s also the only end-to-end solution for all your testing needs — from test creation to execution and analysis.
To learn more, check out check out Perfecto’s Reporting and Analytics to see how to test smarter and work faster.
Smart Reporting and Analytics
DevOps Chief Evangelist & Sr. Director at Perforce Software, Perfecto
Eran Kinsbruner is a person overflowing with ideas and inspiration, beyond that, he makes them happen. He is a best-selling author, continuous-testing and DevOps thought-leader, patent-holding inventor (test exclusion automated mechanisms for mobile J2ME testing), international speaker, and blogger.
With a background of over 20 years of experience in development and testing, Eran empowers clients to create products that their customers love, igniting real results for their companies.