The 2022 State of Test Automation Report

Introduction

Welcome to the 2022 State of Test Automation Report

Now in its third year of publication, Perfecto’s annual State of Test Automation report checks the pulse of test automation - including trends in testing, automation rates, and testing concerns and priorities for 2022. 

To learn more about the challenges and focuses of testing teams today, we surveyed over 400 leading digital enterprises across industries. Here’s what we found, and where testing is going this year. 

Key Takeaways

• Test stability and test coverage are still challenging for teams: Keeping up with high-value test suites is particularly difficult as modern apps and the market continuously change. 
• Test failure analysis is even more time-consuming: While manual testing still reigns as the most time-consuming aspect of testing, practitioners are increasingly seeing test failure analysis as a major bottleneck.  
• Shift-left testing is expanding: In 2022, organizations are looking to expand shift-left testing to API, performance, and accessibility testing. 
• Top performers are embracing continuous testing: With higher levels of automation within the release cycle, leading organizations are vetting BDD and scriptless solutions, as well as open-source technologies to support testing efforts. 

Test Automation Stability & Coverage Challenge Teams Most

This year, the most painful challenges for testing teams are test stability and test coverage.

Testing Challenges

With the release of advanced technologies, in addition to constant browser and OS updates, maintaining sufficient test coverage is increasingly difficult.  

Frequent changes in modern apps and the market also impact test stability. While there are other reasons why tests might fail, tests can merely fail due to a new device or browser version that uncovers a bug in the application software. 

Test Automation Coverage Solutions 

Here are some ways to make sure that your test coverage is up to date.  

1. Leverage market data and keep your lab up to date and aligned with your own customer usage data. 

• Use calendars. 
• Follow market events. 
• Use web traffic analysis. 
• Match coverage to historical defects. 

2. Continuously maintain, validate, and modify your test automation suites.  

• Keep your suite lean and valuable. 
• Never repeat failing tests without debugging. 
• Ensure your CI BAT and regression suites are constantly updated. 

3. Spread your test automation executions throughout the pipeline based on… 

• Software iteration scope. 
• Regression needs, parallel testing for velocity 
• Target features and platforms. 
• Phase in the CI/CD pipeline (use test pyramid). 
• Persona, skillset, and tools. 

Related Reading: Test Coverage Guide to Update Your Testing Strategy

Get Your Guide

Test Automation Visibility Recommendations

Making team members aware of relevant issues early on will help resolve them quickly and increase productivity. 

Breaking It Down By Persona 

Developer 

• Show me my build status. 
• Point me to the exact breaking point. 
• I’d like to get data using Jira, Jenkins, and Slack. 

SDET  

• Help me optimize quality using tags and custom fields. 
• Maximize context during test authoring. 
• Work within my IDE/FW. 

QA Managers & Executives 

• Have the quality visibility to support decisions during the SDLC. 
• Understand where problems are to distribute across the right people/teams. 
• Understand trends and challenges over time. 

Agile/Feature Teams 

• Cross-feature team communication through Slack, Jira, and custom reports. 
• DevOps quality visibility per feature team and post integration phase. 

Test Failure Analysis Increasingly Time-Consuming

Teams want to shift their tests left to minimize the more time-consuming aspects of testing. This year, test failure analysis joins manual testing as an especially tedious part of the software testing lifecycle.

What is the most time-consuming activity for you within a test cycle?

There are four main reasons why tests fail: 

1. Environment issues. 
2. Test issues.  
3. Issues with test data.  
4. Actual defects. 

According to Perfecto’s data, no more than 4% of failures end up being actual defects. Yet defects are the most important failures to analyze when looking through testing reports. Organizations that do not have coordinated reporting methods end up spending more time than necessary on backlog looking at the other reasons why tests fail.  

Perfecto’s various reporting features, particularly its latest Test Automation Insights capability, drive an optimized pipeline by categorizing failures within the Perfecto reporting dashboard, as well as sharing the organization’s most common issues and failure trends.  

Top organizations leverage these types of insights, as well as other methods such as heatmaps, to cut through the noise and spend no more than 30 minutes evaluating and resolving issues behind failed test cases. 

Shift-left testing also helps teams address the most time-consuming aspects of testing, due to the following benefits: 

• It is more cost-effective to reveal defects earlier (Fail fast/Fix fast). 
• It supports the goals of Agile/DevOps (Releasing Faster). 
• Easier to maintain code and quality. 
• Covers more functional and non-functional areas early on (Security, accessibility, performance, API). 

Related Reading: Test Reporting: What It Is & How to Make It Work for Continuous Testing

Read Blog

Mobile Testing Remains Highly Important

Over 60% of respondents said native mobile and mobile web apps are very important to their testing strategy. As in previous years, this revelation comes as no surprise. 

One a scale of 1 (low) to 5 (high), please rate how important mobile native and mobile web apps are in your overall testing strategy.

With the rise of new mobile technologies like 5G and foldables, mobile and mobile web application testing are increasingly complex. As mobile apps become more advanced and mobile phone releases happen every few months, mobile test automation feels like a moving target. 

Related Reading: The Complete Guide to Mobile Automation Testing

Download eBook

BDD & Continuous Testing On the Rise

With most organizations adopting some sort of automated software testing (66% of respondents still automate fewer than 50% of tests), the adoption of BDD and continuous testing is growing within the industry as well.

Testing Practices Adoption and Maturity

Continuous testing is mostly done by two types of teams: an internal QA team that supports multiple agile teams, or dedicated QA resources that are part of a feature team or squad. These resources support their organization’s release cadence, which averages between weekly and monthly.

Yet, when teams continue with weekly or monthly release cadences without high test automation percentage rates, there is an increased risk of defects escaping to production.

BDD testing offers a wide range of benefits, including its business-focused perspective, the simplicity of writing Gherkin tests in simple English, and its resultant ability to help increase collaboration across teams.

Making BDD part of your continuous testing process will help your team give stakeholders fast and easy-to-understand feedback about how your application is functioning.

Related Reading: The Top 3 Roadblocks to Sustainable Continuous Testing

Download eBook

Continuous testing and BDD are very well adopted within the industry, which help support frequent release cadences.

Shift-Left Testing Is Expanding

In 2022, organizations are looking to expand shift-left testing to non-functional testing, particularly API, performance, and accessibility testing.

Which Test Automation Types Are Your Main Focus Area in 2022?

Which Trends Impacting Testing Are You Looking to Invest in This Year?

While shifting testing left will remain critical to effective functional testing, shifting more aspects of testing left will be especially important in the new year.

To date, most organizations do not worry about areas like performance, accessibility, or security testing until the end of each sprint, or worse, only when something goes wrong. To address issues presented by performance and accessibility testing, organizations are looking for ways to incorporate these testing types into the sprints themselves.

Shift-left testing for non-functional tests (e.g., performance, API, accessibility, security) is becoming a major focus for organizations in 2022.

46% Investing in Commercial Test Creation Solutions

The interest in commercial test creation tools is on the rise. These solutions are especially timely as shifting left non-functional tests like performance and accessibility testing is becoming a major focus.

Which Testing Tools Are Organizations Investing in?

Combining continuous testing platforms that support both functional and non-functional testing helps make web and mobile testing end-to-end a simpler and more achievable process.

Commercial platforms like Perfecto and BlazeMeter, both under the Perforce umbrella, offer a unique cloud-based environment to create, execute, and analyze all testing types, such as API, functional, non-functional, exploratory, and unit. Using platforms like these together covers the entire testing pyramid at a maximum scale.

Related Reading: What Can Perfecto & BlazeMeter Do for You?

Read Blog

Commercial testing tools that support functional testing, as well as performance, accessibility, and API testing, are growing in popularity.

Teams Seek New Tools & Trends in 2022

Cypress and Scriptless testing are top of mind in 2022. 41% of respondents have an interest in using Cypress, while 39% of respondents have an interest in scriptless or intelligent test automation solutions this year.

Playwright is also emerging as an area of interest for test automation frameworks.

Which frameworks are top of mind to you in 2022?

41% of companies are interested in using Cypress, while 39% are interested in using codeless solutions this year.

AI/ML a Top Priority for 2022

Incorporating AI and machine learning into the testing process is a major priority for 2022. There are many benefits to AI in testing, including:

• Making processes more efficient.
• Improving decision-making.
• Enhancing existing products and services.

Process efficiency tops the list of benefits achieved with AI

Note: Orange dotted lines represent the average of each dimension.
Source: Deloitte, State of AI in the Enterprise, 2020

From test creating to test reporting, adding AI-based technology to these processes helps teams test earlier, more productively, and more intelligently.

Related Reading: The Current State & Future Trends of AI in Software Testing

Read Blog

39% of companies are interested in adding AI/ML to their testing this year.

The Shift from Manual to Automated Testing Is Ongoing

Companies are prioritizing moving from manual to automated testing as much as possible.

What is Your Top Priority Around Software Testing in 2022?

The biggest advantage of automation testing over manual testing is that it allows for more testing in less time. It increases productivity and expands how much you can test.

While there will always be a need for manual testing, automating the right tests will help testers execute more test cases in less time, improve test coverage, and keep up with the pressure of go-to-market needs.

Related Reading: Manual Testing vs. Automated Testing

Read Blog

30% of organizations are prioritizing the shift from manual to automated testing in 2022.

A Year in Review & What’s to Come in 2022

As the world recovers from the havoc of COVID-19 while simultaneously dealing with new waves, 2022 is both a time of transition and expansion for software testing and development. 

While some of the concerns and priorities in 2022 are the same as previous years, teams are looking for new ways to mature their testing and leverage new technologies.  

Scaled Test Automation That Shifts Left

With more testing done on digital platforms, teams show a greater increase in test automation investment. In 2021, teams not only grew their mobile and web app test suites, but also adopted additional technologies to serve different personas within the same team.

Frameworks such as Cypress and Flutter, together with more BDD adoption, have seen tremendous usage as developers and testers take on more testing responsibilities. This is not going to change going forward as digital winners will continue with such investments.

Additionally, the growing emphasis on shifting testing left will place more focus on performance, accessibility, and API test automation across platforms. These types of testing have become top of mind for digital businesses serving all types of users, as well as eliminating manual testing activities that happen too late in the development process.

Optimized Test Failure Analysis 

As teams look to improve testing efficiency, they will look to the aspects of the testing process that are the most time-consuming. The analysis of test failures has become increasingly difficult, due to the amount of data and results that practitioners must comb through daily.

Top organizations will look for ways to cut through the noise of too much data to keep test failure analysis both helpful and manageable. Organizations will work to shorten the amount of time spent on test failure analysis with tools such as heatmaps, insights from reports, and knowledge of common failure reasons.

AI/ML will also continue to play a role in optimizing test failure analysis and improving other areas of testing. Intelligent test automation is still a major area of interest in 2022 and will continue to play an important role in testing moving forward.  

Expanded Test Coverage & Stability

Organizations will also continue to look for ways to expand their test stability and coverage in 2022. By keeping test suites lean and valuable, while leveraging testing labs that support the latest devices, browsers, and OSs, keeping up with market changes can be much simpler.

We also see increased collaboration amongst team members to maintain test stability and resolve issues quickly. By increasing visibility and tailoring those insights to the responsibilities of each persona involved in testing (tester, developer, SDET, and agile/feature teams), will increase productivity and efficiency on all fronts.

About Perfecto

Perfecto by Perforce is the leading enterprise-grade cloud platform for mobile and desktop web application testing. Perfecto’s fully hybrid, cloud platform enables software test engineers and developers to develop, execute and analyze their results at scale. Not only can customers choose between public and private dedicated environments, but they can also gain immediate access to any real and virtual mobile and web platforms to best fit their needs. The platform is fully open and integrated to support much of the CI/CD tool stack and all leading test automation frameworks including Selenium, Appium, Espresso, XCUITest, Cypress and more. Used by some of the largest, global enterprises including Lloyds Bank, Allstate, Verizon, Wayfair, Ford, IBM and more, companies of nearly every vertical depend on Perfecto to improve quality, accelerate releases, and scale automation. For more information about Perfecto, visit perfecto.io. 

Try Perfecto