Introduction

Welcome to the 2022 State of Test Automation Report

Now in its third year of publication, Perfecto’s annual State of Test Automation report checks the pulse of test automation - including trends in testing, automation rates, and testing concerns and priorities for 2022. 

To learn more about the challenges and focuses of testing teams today, we surveyed over 400 leading digital enterprises across industries. Here’s what we found, and where testing is going this year. 

Key Takeaways

  • Test stability and test coverage are still challenging for teams: Keeping up with high-value test suites is particularly difficult as modern apps and the market continuously change. 
  • Test failure analysis is even more time-consuming: While manual testing still reigns as the most time-consuming aspect of testing, practitioners are increasingly seeing test failure analysis as a major bottleneck.  
  • Shift-left testing is expanding: In 2022, organizations are looking to expand shift-left testing to API, performance, and accessibility testing. 
  • Top performers are embracing continuous testing: With higher levels of automation within the release cycle, leading organizations are vetting BDD and scriptless solutions, as well as open-source technologies to support testing efforts. 
Cloud Automation

Test Automation Stability & Coverage Challenge Teams Most

This year, the most painful challenges for testing teams are test stability and test coverage.

Testing Challenges

biggest challenge

With the release of advanced technologies, in addition to constant browser and OS updates, maintaining sufficient test coverage is increasingly difficult.  

Frequent changes in modern apps and the market also impact test stability. While there are other reasons why tests might fail, tests can merely fail due to a new device or browser version that uncovers a bug in the application software. 

Test Automation Coverage Solutions 

Here are some ways to make sure that your test coverage is up to date.  

1. Leverage market data and keep your lab up to date and aligned with your own customer usage data. 

  • Use calendars. 
  • Follow market events. 
  • Use web traffic analysis. 
  • Match coverage to historical defects. 

2. Continuously maintain, validate, and modify your test automation suites.  

  • Keep your suite lean and valuable. 
  • Never repeat failing tests without debugging. 
  • Ensure your CI BAT and regression suites are constantly updated. 

3. Spread your test automation executions throughout the pipeline based on… 

  • Software iteration scope. 
  • Regression needs, parallel testing for velocity 
  • Target features and platforms. 
  • Phase in the CI/CD pipeline (use test pyramid). 
  • Persona, skillset, and tools. 

Related Reading: Test Coverage Guide to Update Your Testing Strategy

Get Your Guide

Test Automation Visibility Recommendations

Making team members aware of relevant issues early on will help resolve them quickly and increase productivity. 

Breaking It Down By Persona 

Developer 
  • Show me my build status. 
  • Point me to the exact breaking point. 
  • I’d like to get data using Jira, Jenkins, and Slack. 
SDET  
  • Help me optimize quality using tags and custom fields. 
  • Maximize context during test authoring. 
  • Work within my IDE/FW. 
QA Managers & Executives 
  • Have the quality visibility to support decisions during the SDLC. 
  • Understand where problems are to distribute across the right people/teams. 
  • Understand trends and challenges over time. 
Agile/Feature Teams 
  • Cross-feature team communication through Slack, Jira, and custom reports. 
  • DevOps quality visibility per feature team and post integration phase. 
Keeping your test suite lean and up to date while improving visibility across your testing team will improve test coverage and stability. 

Test Failure Analysis Increasingly Time-Consuming

Teams want to shift their tests left to minimize the more time-consuming aspects of testing. This year, test failure analysis joins manual testing as an especially tedious part of the software testing lifecycle.

What is the most time-consuming activity for you within a test cycle?

time consuming activity

There are four main reasons why tests fail: 

  1. Environment issues. 
  2. Test issues.  
  3. Issues with test data.  
  4. Actual defects. 

According to Perfecto’s data, no more than 4% of failures end up being actual defects. Yet defects are the most important failures to analyze when looking through testing reports. Organizations that do not have coordinated reporting methods end up spending more time than necessary on backlog looking at the other reasons why tests fail.  

Perfecto’s various reporting features, particularly its latest Test Automation Insights capability, drive an optimized pipeline by categorizing failures within the Perfecto reporting dashboard, as well as sharing the organization’s most common issues and failure trends.  

Top organizations leverage these types of insights, as well as other methods such as heatmaps, to cut through the noise and spend no more than 30 minutes evaluating and resolving issues behind failed test cases. 

Shift-left testing also helps teams address the most time-consuming aspects of testing, due to the following benefits: 

  • It is more cost-effective to reveal defects earlier (Fail fast/Fix fast). 
  • It supports the goals of Agile/DevOps (Releasing Faster). 
  • Easier to maintain code and quality. 
  • Covers more functional and non-functional areas early on (Security, accessibility, performance, API). 

Related Reading: Test Reporting: What It Is & How to Make It Work for Continuous Testing

Read Blog

Shift-left testing can address the most time-consuming aspects of testing.

Mobile Testing Remains Highly Important

Over 60% of respondents said native mobile and mobile web apps are very important to their testing strategy. As in previous years, this revelation comes as no surprise. 

One a scale of 1 (low) to 5 (high), please rate how important mobile native and mobile web apps are in your overall testing strategy.

mobile app importance

With the rise of new mobile technologies like 5G and foldables, mobile and mobile web application testing are increasingly complex. As mobile apps become more advanced and mobile phone releases happen every few months, mobile test automation feels like a moving target. 

Related Reading: The Complete Guide to Mobile Automation Testing

Download eBook

Mobile app testing and keeping up with the latest mobile tech & trends is more critical than ever in 2022.

BDD & Continuous Testing On the Rise

With most organizations adopting some sort of automated software testing (66% of respondents still automate fewer than 50% of tests), the adoption of BDD and continuous testing is growing within the industry as well.

Testing Practices Adoption and Maturity

adoption maturity

Continuous testing is mostly done by two types of teams: an internal QA team that supports multiple agile teams, or dedicated QA resources that are part of a feature team or squad. These resources support their organization’s release cadence, which averages between weekly and monthly.

Yet, when teams continue with weekly or monthly release cadences without high test automation percentage rates, there is an increased risk of defects escaping to production.

BDD testing offers a wide range of benefits, including its business-focused perspective, the simplicity of writing Gherkin tests in simple English, and its resultant ability to help increase collaboration across teams.

Making BDD part of your continuous testing process will help your team give stakeholders fast and easy-to-understand feedback about how your application is functioning.

Related Reading: The Top 3 Roadblocks to Sustainable Continuous Testing

Download eBook

Continuous testing and BDD are very well adopted within the industry, which help support frequent release cadences.

Shift-Left Testing Is Expanding

In 2022, organizations are looking to expand shift-left testing to non-functional testing, particularly API, performance, and accessibility testing.

Which Test Automation Types Are Your Main Focus Area in 2022?

main focus area

Which Trends Impacting Testing Are You Looking to Invest in This Year?

investing in trends

While shifting testing left will remain critical to effective functional testing, shifting more aspects of testing left will be especially important in the new year.

To date, most organizations do not worry about areas like performance, accessibility, or security testing until the end of each sprint, or worse, only when something goes wrong. To address issues presented by performance and accessibility testing, organizations are looking for ways to incorporate these testing types into the sprints themselves.

Shift-left testing for non-functional tests (e.g., performance, API, accessibility, security) is becoming a major focus for organizations in 2022.

46% Investing in Commercial Test Creation Solutions

The interest in commercial test creation tools is on the rise. These solutions are especially timely as shifting left non-functional tests like performance and accessibility testing is becoming a major focus.

Which Testing Tools Are Organizations Investing in?

tools investing in

Combining continuous testing platforms that support both functional and non-functional testing helps make web and mobile testing end-to-end a simpler and more achievable process.

Commercial platforms like Perfecto and BlazeMeter, both under the Perforce umbrella, offer a unique cloud-based environment to create, execute, and analyze all testing types, such as API, functional, non-functional, exploratory, and unit. Using platforms like these together covers the entire testing pyramid at a maximum scale.

Related Reading: What Can Perfecto & BlazeMeter Do for You?

Read Blog

Commercial testing tools that support functional testing, as well as performance, accessibility, and API testing, are growing in popularity.