IntroductionWelcome to the 2022 State of Test Automation ReportNow in its third year of publication, Perfecto’s annual State of Test Automation report checks the pulse of test automation - including trends in testing, automation rates, and testing concerns and priorities for 2022. To learn more about the challenges and focuses of testing teams today, we surveyed over 400 leading digital enterprises across industries. Here’s what we found, and where testing is going this year. Key TakeawaysTest stability and test coverage are still challenging for teams: Keeping up with high-value test suites is particularly difficult as modern apps and the market continuously change. Test failure analysis is even more time-consuming: While manual testing still reigns as the most time-consuming aspect of testing, practitioners are increasingly seeing test failure analysis as a major bottleneck. Shift-left testing is expanding: In 2022, organizations are looking to expand shift-left testing to API, performance, and accessibility testing. Top performers are embracing continuous testing: With higher levels of automation within the release cycle, leading organizations are vetting BDD and scriptless solutions, as well as open-source technologies to support testing efforts. Test Automation Stability & Coverage Challenge Teams MostThis year, the most painful challenges for testing teams are test stability and test coverage.Testing ChallengesWith the release of advanced technologies, in addition to constant browser and OS updates, maintaining sufficient test coverage is increasingly difficult. Frequent changes in modern apps and the market also impact test stability. While there are other reasons why tests might fail, tests can merely fail due to a new device or browser version that uncovers a bug in the application software. Test Automation Coverage Solutions Here are some ways to make sure that your test coverage is up to date. 1. Leverage market data and keep your lab up to date and aligned with your own customer usage data. Use calendars. Follow market events. Use web traffic analysis. Match coverage to historical defects. 2. Continuously maintain, validate, and modify your test automation suites. Keep your suite lean and valuable. Never repeat failing tests without debugging. Ensure your CI BAT and regression suites are constantly updated. 3. Spread your test automation executions throughout the pipeline based on… Software iteration scope. Regression needs, parallel testing for velocity Target features and platforms. Phase in the CI/CD pipeline (use test pyramid). Persona, skillset, and tools. Related Reading: Test Coverage Guide to Update Your Testing StrategyGet Your GuideTest Automation Visibility RecommendationsMaking team members aware of relevant issues early on will help resolve them quickly and increase productivity. Breaking It Down By Persona Developer Show me my build status. Point me to the exact breaking point. I’d like to get data using Jira, Jenkins, and Slack. SDET Help me optimize quality using tags and custom fields. Maximize context during test authoring. Work within my IDE/FW. QA Managers & Executives Have the quality visibility to support decisions during the SDLC. Understand where problems are to distribute across the right people/teams. Understand trends and challenges over time. Agile/Feature Teams Cross-feature team communication through Slack, Jira, and custom reports. DevOps quality visibility per feature team and post integration phase. Keeping your test suite lean and up to date while improving visibility across your testing team will improve test coverage and stability. Test Failure Analysis Increasingly Time-ConsumingTeams want to shift their tests left to minimize the more time-consuming aspects of testing. This year, test failure analysis joins manual testing as an especially tedious part of the software testing lifecycle.What is the most time-consuming activity for you within a test cycle?There are four main reasons why tests fail: Environment issues. Test issues. Issues with test data. Actual defects. According to Perfecto’s data, no more than 4% of failures end up being actual defects. Yet defects are the most important failures to analyze when looking through testing reports. Organizations that do not have coordinated reporting methods end up spending more time than necessary on backlog looking at the other reasons why tests fail. Perfecto’s various reporting features, particularly its latest Test Automation Insights capability, drive an optimized pipeline by categorizing failures within the Perfecto reporting dashboard, as well as sharing the organization’s most common issues and failure trends. Top organizations leverage these types of insights, as well as other methods such as heatmaps, to cut through the noise and spend no more than 30 minutes evaluating and resolving issues behind failed test cases. Shift-left testing also helps teams address the most time-consuming aspects of testing, due to the following benefits: It is more cost-effective to reveal defects earlier (Fail fast/Fix fast). It supports the goals of Agile/DevOps (Releasing Faster). Easier to maintain code and quality. Covers more functional and non-functional areas early on (Security, accessibility, performance, API). Related Reading: Test Reporting: What It Is & How to Make It Work for Continuous TestingRead BlogShift-left testing can address the most time-consuming aspects of testing.Mobile Testing Remains Highly ImportantOver 60% of respondents said native mobile and mobile web apps are very important to their testing strategy. As in previous years, this revelation comes as no surprise. One a scale of 1 (low) to 5 (high), please rate how important mobile native and mobile web apps are in your overall testing strategy.With the rise of new mobile technologies like 5G and foldables, mobile and mobile web application testing are increasingly complex. As mobile apps become more advanced and mobile phone releases happen every few months, mobile test automation feels like a moving target. Related Reading: The Complete Guide to Mobile Automation TestingDownload eBookMobile app testing and keeping up with the latest mobile tech & trends is more critical than ever in 2022.BDD & Continuous Testing On the RiseWith most organizations adopting some sort of automated software testing (66% of respondents still automate fewer than 50% of tests), the adoption of BDD and continuous testing is growing within the industry as well.Testing Practices Adoption and MaturityContinuous testing is mostly done by two types of teams: an internal QA team that supports multiple agile teams, or dedicated QA resources that are part of a feature team or squad. These resources support their organization’s release cadence, which averages between weekly and monthly.Yet, when teams continue with weekly or monthly release cadences without high test automation percentage rates, there is an increased risk of defects escaping to production.BDD testing offers a wide range of benefits, including its business-focused perspective, the simplicity of writing Gherkin tests in simple English, and its resultant ability to help increase collaboration across teams.Making BDD part of your continuous testing process will help your team give stakeholders fast and easy-to-understand feedback about how your application is functioning.Related Reading: The Top 3 Roadblocks to Sustainable Continuous TestingDownload eBookContinuous testing and BDD are very well adopted within the industry, which help support frequent release cadences.Shift-Left Testing Is ExpandingIn 2022, organizations are looking to expand shift-left testing to non-functional testing, particularly API, performance, and accessibility testing.Which Test Automation Types Are Your Main Focus Area in 2022?Which Trends Impacting Testing Are You Looking to Invest in This Year?While shifting testing left will remain critical to effective functional testing, shifting more aspects of testing left will be especially important in the new year.To date, most organizations do not worry about areas like performance, accessibility, or security testing until the end of each sprint, or worse, only when something goes wrong. To address issues presented by performance and accessibility testing, organizations are looking for ways to incorporate these testing types into the sprints themselves.Shift-left testing for non-functional tests (e.g., performance, API, accessibility, security) is becoming a major focus for organizations in 2022.46% Investing in Commercial Test Creation SolutionsThe interest in commercial test creation tools is on the rise. These solutions are especially timely as shifting left non-functional tests like performance and accessibility testing is becoming a major focus.Which Testing Tools Are Organizations Investing in?Combining continuous testing platforms that support both functional and non-functional testing helps make web and mobile testing end-to-end a simpler and more achievable process.Commercial platforms like Perfecto and BlazeMeter, both under the Perforce umbrella, offer a unique cloud-based environment to create, execute, and analyze all testing types, such as API, functional, non-functional, exploratory, and unit. Using platforms like these together covers the entire testing pyramid at a maximum scale.Related Reading: What Can Perfecto & BlazeMeter Do for You?Read BlogCommercial testing tools that support functional testing, as well as performance, accessibility, and API testing, are growing in popularity.Teams Seek New Tools & Trends in 2022Cypress and Scriptless testing are top of mind in 2022. 41% of respondents have an interest in using Cypress, while 39% of respondents have an interest in scriptless or intelligent test automation solutions this year.Playwright is also emerging as an area of interest for test automation frameworks.Which frameworks are top of mind to you in 2022?41% of companies are interested in using Cypress, while 39% are interested in using codeless solutions this year.AI/ML a Top Priority for 2022Incorporating AI and machine learning into the testing process is a major priority for 2022. There are many benefits to AI in testing, including:Making processes more efficient.Improving decision-making.Enhancing existing products and services.Process efficiency tops the list of benefits achieved with AINote: Orange dotted lines represent the average of each dimension.Source: Deloitte, State of AI in the Enterprise, 2020From test creating to test reporting, adding AI-based technology to these processes helps teams test earlier, more productively, and more intelligently.Related Reading: The Current State & Future Trends of AI in Software TestingRead Blog39% of companies are interested in adding AI/ML to their testing this year.The Shift from Manual to Automated Testing Is OngoingCompanies are prioritizing moving from manual to automated testing as much as possible.What is Your Top Priority Around Software Testing in 2022?