View all web browser and mobile devices available in our cloud-based test lab.
Now in its third year of publication, Perfecto’s annual State of Test Automation report checks the pulse of test automation - including trends in testing, automation rates, and testing concerns and priorities for 2022.
To learn more about the challenges and focuses of testing teams today, we surveyed over 400 leading digital enterprises across industries. Here’s what we found, and where testing is going this year.
This year, the most painful challenges for testing teams are test stability and test coverage.
With the release of advanced technologies, in addition to constant browser and OS updates, maintaining sufficient test coverage is increasingly difficult.
Frequent changes in modern apps and the market also impact test stability. While there are other reasons why tests might fail, tests can merely fail due to a new device or browser version that uncovers a bug in the application software.
Here are some ways to make sure that your test coverage is up to date.
1. Leverage market data and keep your lab up to date and aligned with your own customer usage data.
2. Continuously maintain, validate, and modify your test automation suites.
3. Spread your test automation executions throughout the pipeline based on…
Related Reading: Test Coverage Guide to Update Your Testing Strategy
Get Your Guide
Making team members aware of relevant issues early on will help resolve them quickly and increase productivity.
Teams want to shift their tests left to minimize the more time-consuming aspects of testing. This year, test failure analysis joins manual testing as an especially tedious part of the software testing lifecycle.
There are four main reasons why tests fail:
According to Perfecto’s data, no more than 4% of failures end up being actual defects. Yet defects are the most important failures to analyze when looking through testing reports. Organizations that do not have coordinated reporting methods end up spending more time than necessary on backlog looking at the other reasons why tests fail.
Perfecto’s various reporting features, particularly its latest Test Automation Insights capability, drive an optimized pipeline by categorizing failures within the Perfecto reporting dashboard, as well as sharing the organization’s most common issues and failure trends.
Top organizations leverage these types of insights, as well as other methods such as heatmaps, to cut through the noise and spend no more than 30 minutes evaluating and resolving issues behind failed test cases.
Shift-left testing also helps teams address the most time-consuming aspects of testing, due to the following benefits:
Related Reading: Test Reporting: What It Is & How to Make It Work for Continuous Testing
Over 60% of respondents said native mobile and mobile web apps are very important to their testing strategy. As in previous years, this revelation comes as no surprise.
With the rise of new mobile technologies like 5G and foldables, mobile and mobile web application testing are increasingly complex. As mobile apps become more advanced and mobile phone releases happen every few months, mobile test automation feels like a moving target.
Related Reading: The Complete Guide to Mobile Automation Testing
With most organizations adopting some sort of automated software testing (66% of respondents still automate fewer than 50% of tests), the adoption of BDD and continuous testing is growing within the industry as well.
Continuous testing is mostly done by two types of teams: an internal QA team that supports multiple agile teams, or dedicated QA resources that are part of a feature team or squad. These resources support their organization’s release cadence, which averages between weekly and monthly.
Yet, when teams continue with weekly or monthly release cadences without high test automation percentage rates, there is an increased risk of defects escaping to production.
BDD testing offers a wide range of benefits, including its business-focused perspective, the simplicity of writing Gherkin tests in simple English, and its resultant ability to help increase collaboration across teams.
Making BDD part of your continuous testing process will help your team give stakeholders fast and easy-to-understand feedback about how your application is functioning.
Related Reading: The Top 3 Roadblocks to Sustainable Continuous Testing
In 2022, organizations are looking to expand shift-left testing to non-functional testing, particularly API, performance, and accessibility testing.
While shifting testing left will remain critical to effective functional testing, shifting more aspects of testing left will be especially important in the new year.
To date, most organizations do not worry about areas like performance, accessibility, or security testing until the end of each sprint, or worse, only when something goes wrong. To address issues presented by performance and accessibility testing, organizations are looking for ways to incorporate these testing types into the sprints themselves.
The interest in commercial test creation tools is on the rise. These solutions are especially timely as shifting left non-functional tests like performance and accessibility testing is becoming a major focus.
Combining continuous testing platforms that support both functional and non-functional testing helps make web and mobile testing end-to-end a simpler and more achievable process.
Commercial platforms like Perfecto and BlazeMeter, both under the Perforce umbrella, offer a unique cloud-based environment to create, execute, and analyze all testing types, such as API, functional, non-functional, exploratory, and unit. Using platforms like these together covers the entire testing pyramid at a maximum scale.
Related Reading: What Can Perfecto & BlazeMeter Do for You?
Cypress and Scriptless testing are top of mind in 2022. 41% of respondents have an interest in using Cypress, while 39% of respondents have an interest in scriptless or intelligent test automation solutions this year.
Playwright is also emerging as an area of interest for test automation frameworks.
Incorporating AI and machine learning into the testing process is a major priority for 2022. There are many benefits to AI in testing, including:
Note: Orange dotted lines represent the average of each dimension.Source: Deloitte, State of AI in the Enterprise, 2020
From test creating to test reporting, adding AI-based technology to these processes helps teams test earlier, more productively, and more intelligently.
Related Reading: The Current State & Future Trends of AI in Software Testing
Companies are prioritizing moving from manual to automated testing as much as possible.
The biggest advantage of automation testing over manual testing is that it allows for more testing in less time. It increases productivity and expands how much you can test.
While there will always be a need for manual testing, automating the right tests will help testers execute more test cases in less time, improve test coverage, and keep up with the pressure of go-to-market needs.
Related Reading: Manual Testing vs. Automated Testing
As the world recovers from the havoc of COVID-19 while simultaneously dealing with new waves, 2022 is both a time of transition and expansion for software testing and development.
While some of the concerns and priorities in 2022 are the same as previous years, teams are looking for new ways to mature their testing and leverage new technologies.
With more testing done on digital platforms, teams show a greater increase in test automation investment. In 2021, teams not only grew their mobile and web app test suites, but also adopted additional technologies to serve different personas within the same team.
Frameworks such as Cypress and Flutter, together with more BDD adoption, have seen tremendous usage as developers and testers take on more testing responsibilities. This is not going to change going forward as digital winners will continue with such investments.
Additionally, the growing emphasis on shifting testing left will place more focus on performance, accessibility, and API test automation across platforms. These types of testing have become top of mind for digital businesses serving all types of users, as well as eliminating manual testing activities that happen too late in the development process.
As teams look to improve testing efficiency, they will look to the aspects of the testing process that are the most time-consuming. The analysis of test failures has become increasingly difficult, due to the amount of data and results that practitioners must comb through daily.
Top organizations will look for ways to cut through the noise of too much data to keep test failure analysis both helpful and manageable. Organizations will work to shorten the amount of time spent on test failure analysis with tools such as heatmaps, insights from reports, and knowledge of common failure reasons.
AI/ML will also continue to play a role in optimizing test failure analysis and improving other areas of testing. Intelligent test automation is still a major area of interest in 2022 and will continue to play an important role in testing moving forward.
Organizations will also continue to look for ways to expand their test stability and coverage in 2022. By keeping test suites lean and valuable, while leveraging testing labs that support the latest devices, browsers, and OSs, keeping up with market changes can be much simpler.
We also see increased collaboration amongst team members to maintain test stability and resolve issues quickly. By increasing visibility and tailoring those insights to the responsibilities of each persona involved in testing (tester, developer, SDET, and agile/feature teams), will increase productivity and efficiency on all fronts.
Perfecto by Perforce is the leading enterprise-grade cloud platform for mobile and desktop web application testing. Perfecto’s fully hybrid, cloud platform enables software test engineers and developers to develop, execute and analyze their results at scale. Not only can customers choose between public and private dedicated environments, but they can also gain immediate access to any real and virtual mobile and web platforms to best fit their needs. The platform is fully open and integrated to support much of the CI/CD tool stack and all leading test automation frameworks including Selenium, Appium, Espresso, XCUITest, Cypress and more. Used by some of the largest, global enterprises including Lloyds Bank, Allstate, Verizon, Wayfair, Ford, IBM and more, companies of nearly every vertical depend on Perfecto to improve quality, accelerate releases, and scale automation. For more information about Perfecto, visit perfecto.io.