View all web browser and mobile devices available in our cloud-based test lab.
Now in its third year of publication, Perfecto’s annual State of Test Automation report checks the pulse of test automation - including trends in testing, automation rates, and testing concerns and priorities for 2022.
To learn more about the challenges and focuses of testing teams today, we surveyed over 400 leading digital enterprises across industries. Here’s what we found, and where testing is going this year.
This year, the most painful challenges for testing teams are test stability and test coverage.
With the release of advanced technologies, in addition to constant browser and OS updates, maintaining sufficient test coverage is increasingly difficult.
Frequent changes in modern apps and the market also impact test stability. While there are other reasons why tests might fail, tests can merely fail due to a new device or browser version that uncovers a bug in the application software.
Here are some ways to make sure that your test coverage is up to date.
1. Leverage market data and keep your lab up to date and aligned with your own customer usage data.
2. Continuously maintain, validate, and modify your test automation suites.
3. Spread your test automation executions throughout the pipeline based on…
Related Reading: Test Coverage Guide to Update Your Testing Strategy
Get Your Guide
Making team members aware of relevant issues early on will help resolve them quickly and increase productivity.
Teams want to shift their tests left to minimize the more time-consuming aspects of testing. This year, test failure analysis joins manual testing as an especially tedious part of the software testing lifecycle.
There are four main reasons why tests fail:
According to Perfecto’s data, no more than 4% of failures end up being actual defects. Yet defects are the most important failures to analyze when looking through testing reports. Organizations that do not have coordinated reporting methods end up spending more time than necessary on backlog looking at the other reasons why tests fail.
Perfecto’s various reporting features, particularly its latest Test Automation Insights capability, drive an optimized pipeline by categorizing failures within the Perfecto reporting dashboard, as well as sharing the organization’s most common issues and failure trends.
Top organizations leverage these types of insights, as well as other methods such as heatmaps, to cut through the noise and spend no more than 30 minutes evaluating and resolving issues behind failed test cases.
Shift-left testing also helps teams address the most time-consuming aspects of testing, due to the following benefits:
Related Reading: Test Reporting: What It Is & How to Make It Work for Continuous Testing
Over 60% of respondents said native mobile and mobile web apps are very important to their testing strategy. As in previous years, this revelation comes as no surprise.
With the rise of new mobile technologies like 5G and foldables, mobile and mobile web application testing are increasingly complex. As mobile apps become more advanced and mobile phone releases happen every few months, mobile test automation feels like a moving target.
Related Reading: The Complete Guide to Mobile Automation Testing
With most organizations adopting some sort of automated software testing (66% of respondents still automate fewer than 50% of tests), the adoption of BDD and continuous testing is growing within the industry as well.
Continuous testing is mostly done by two types of teams: an internal QA team that supports multiple agile teams, or dedicated QA resources that are part of a feature team or squad. These resources support their organization’s release cadence, which averages between weekly and monthly.
Yet, when teams continue with weekly or monthly release cadences without high test automation percentage rates, there is an increased risk of defects escaping to production.
BDD testing offers a wide range of benefits, including its business-focused perspective, the simplicity of writing Gherkin tests in simple English, and its resultant ability to help increase collaboration across teams.
Making BDD part of your continuous testing process will help your team give stakeholders fast and easy-to-understand feedback about how your application is functioning.
Related Reading: The Top 3 Roadblocks to Sustainable Continuous Testing
In 2022, organizations are looking to expand shift-left testing to non-functional testing, particularly API, performance, and accessibility testing.
While shifting testing left will remain critical to effective functional testing, shifting more aspects of testing left will be especially important in the new year.
To date, most organizations do not worry about areas like performance, accessibility, or security testing until the end of each sprint, or worse, only when something goes wrong. To address issues presented by performance and accessibility testing, organizations are looking for ways to incorporate these testing types into the sprints themselves.
The interest in commercial test creation tools is on the rise. These solutions are especially timely as shifting left non-functional tests like performance and accessibility testing is becoming a major focus.
Combining continuous testing platforms that support both functional and non-functional testing helps make web and mobile testing end-to-end a simpler and more achievable process.
Commercial platforms like Perfecto and BlazeMeter, both under the Perforce umbrella, offer a unique cloud-based environment to create, execute, and analyze all testing types, such as API, functional, non-functional, exploratory, and unit. Using platforms like these together covers the entire testing pyramid at a maximum scale.
Related Reading: What Can Perfecto & BlazeMeter Do for You?
Cypress and Scriptless testing are top of mind in 2022. 41% of respondents have an interest in using Cypress, while 39% of respondents have an interest in scriptless or intelligent test automation solutions this year.
Playwright is also emerging as an area of interest for test automation frameworks.
Incorporating AI and machine learning into the testing process is a major priority for 2022. There are many benefits to AI in testing, including:
Note: Orange dotted lines represent the average of each dimension.Source: Deloitte, State of AI in the Enterprise, 2020
From test creating to test reporting, adding AI-based technology to these processes helps teams test earlier, more productively, and more intelligently.
Related Reading: The Current State & Future Trends of AI in Software Testing
Companies are prioritizing moving from manual to automated testing as much as possible.