IntroductionWelcome to the 2021 State of Test Automation ReportReady or not, digital platforms were thrust onto center stage in 2020, raising the stakes for high-performing web and mobile apps and making the ability to test remotely a requirement.Where did we land? What’s challenging teams today? What are they focusing on? To find out, we surveyed over 700 leading digital enterprises across industries. Here’s what we found, and where we think testing is going this year.Key TakeawaysTeams focused on more automation, better coverage: Despite link to faster releases, half of respondents automate fewer than 50% of tests. They’re changing that in 2021. Shifting left is shifting into high gear: The movement is correlated with faster releases. Larger organizations lead the pack in adoption. Top performers are addressing the automation skills gap. With higher levels of automation within the release cycle, leading orgs are vetting BDD and scriptless solutions. Leadership, practitioners not on same page: C-Suite and VPs believe they are more automated than practitioners report. Alignment needed for prioritizing 2021 investments. Testing Shifts Into Dev Feature TeamsThis year, the majority of testing was done by internal QA supporting development. Who is responsible for testing in your organization?When companies rely on testing resources within development: They are less likely to partake in the following testing practices: Automated software testing. Continuous testing, integration, and delivery. Behavior-driven development. When companies rely on internal QA supporting dev, they are more likely to… Release daily or weekly. Automate more than 50% of tests. Rely on open source technology. Test at the beginning of each software sprint. Teams who outsource their testing are more likely to struggle with test stability. Teams with both internal QA team and testing resource are the most likely to shift testing left. Agile and DevOps maturity is driving the shift of testers into dev feature teams. This is to reduce the time to create and run tests and also to respond faster within the cycle.Test Automation Creation & Coverage Challenge Teams MostAcross the board, teams find the lack of resources for creating test automation to be the biggest challenge they face, followed closely by proper coverage for test automation scenarios. What do you find most challenging about testing?Breaking It Down By Persona Top challenges vary based on role, too. Top Dev struggles: Lack of prioritization for testing Lacking resources for test automation creation Test flakiness Top QA struggles: Lack of resources for test automation creation Getting the right test coverage Test flakiness Challenges Vary Depending on Automation MaturityTeams automating fewer than 10% of their test cases are: More likely to struggle with lab maintenance and a lack of testing prioritization. More likely to release only once or twice per year. Less likely to practice behavior-driven development. Less likely to achieve continuous testing, integration, and delivery. On the other end of the spectrum, teams who have shifted left or are automating over 50% of their test cases: Are more likely to be challenged by false negatives, test flakiness, and instability. Spend more time on script maintenance, test environment set up, and test failure analysis. More likely to release daily or weekly. Test Automation Creation & Coverage Solutions To better tackle test automation creation, teams need to match test automation tools with the skillsets of team members. This allows even more team members to partake in the test automation creation process. It also allows teams to improve their test scenario coverage. For extended test coverage of platforms, teams can make two improvements: Testing with a commercial cloud-based solution rather than a DIY lab. Building a strategy with data-driven coverage as opposed to risk-based coverage. Related Reading: 5 Things to Consider When We Talk About Test CoverageRead BlogCloud-based solutions and data-driven decisions can help teams boost test coverage. More Teams Are Waiting to Test After a BuildSuccessful organizations are able to fit their test automation within the build process and drive faster feedback. Our research finds that ~70% are still struggling to accomplish that objective. And that requires better cross-team collaboration and alignment. Where does testing fit into your software development lifecycle?To mature DevOps, most checkpoints must be checked automatically. The more human intervention there is, the slower the process will be. An example of a best practice can be something like this. Instead of test engineers waiting for a new build to start their test automation, developers can grant an early access build that has the right “tag” attached to it to allow test automation activities to begin. Related Reading: SDLC Tools For Testing: The Right One For Each PhaseRead BlogMature DevOps requires automated checkpoints. Human intervention slows processes down. Teams who have shifted left are more likely to release daily.COVID-19 & Remote Work Slows Teams DownOnly 9% of those surveyed release daily, which is down from 14 % last year. The challenges of remote testing throughout 2020 appear to be slowing teams down.What Is your typical release cycle?How to Release More The keys to releasing more often depends on the entire software development process from coding, building, integrating, testing, and deploying the code to production. Whenever tasks are broken, or not automated, the entire process slows down and delays the release. While in this research we focus on test automation, more than just that needs to be automated. This needs to be a DevOps team priority and not a single team’s goal. Making the connection between the different players in the pipeline comes through process and technology, and this requires choices based on priorities to be made by management. Related Reading: Top 5 Things Slowing Down Your Testing Life CycleRead BlogTeams who have shifted left are more likely to release daily. Manual Testing Hinders Shift Left InitiativesTeams want to shift their testing left — but manual testing is an obstacle to achieving that. Manual testing will always be a reality to some extent, as it’s recommended to mix manual and exploratory testing as part of each software iteration. However, the more tests that can be automated and shifted left, the better.What are the most time-consuming activities within a test cycle?Almost 10% of survey respondents are still manually testing all of their tests. Another 20% are just getting started with automation. And 47% of teams automate fewer than 50% of their tests. Teams struggling with manual testing are…More likely to automate fewer than 10% of their test cases within a release cycle. More likely to wait to test until after a new build is ready. More likely to release only once or twice per year. More likely to rely on open source technology. Related Reading: How to Transition From Manual Testing to Automated Testing in 7 StepsRead BlogTeams can shift left by minimizing manual testing. By matching tools with skillset needs, business testers and noncoders can participate in the testing process. And earlier delivery of software builds through TDD (test driven development) provide them with more time to automate. Mobile Is HIGHLY Important Over 60% of respondents said native mobile and mobile web apps are very important to their testing strategy. That’s almost a 10% increase from 2020 — clearly mobile is top of mind. How important are mobile native and mobile web apps in your overall testing strategy?This comes as no surprise —81% of US adults own a smartphone. And mobile outranks desktop and tablets for market share. The increased importance of mobile and mobile web apps leads to more complex test automation. Mobile apps have become more advanced and mobile phone releases/functionality are constantly moving targets. Related Reading: The Complete Guide to Mobile Automation TestingRead eBookMobile app testing is more critical than ever in 2021. Teams Seek New Tools & Trends in 2021 Scriptless testing is top of mind in 2021. Over 40% of respondents have an interest in using scriptless/intelligent test automation solutions this year. Scriptless testing is also the top trend respondents are looking to invest in throughout 2021.Other emerging areas of interest include Cypress, Flutter, 5G, and PWAs. Which frameworks are top of mind to you in 2021? Select all that apply.Which trends impacting testing are you looking to invest in this year? Select all that apply. 40% of companies are interested in using codeless solutions this year. 22% Leverage Commercial Labs Today And 26% of teams report using commercial test creation solutions. And almost half of companies are looking to invest in testing tools in 2021. What test automation tools are you leveraging today?Teams using commercial lab vendors are…. More likely to achieve continuous testing, integration, and delivery. More like to practice behavior-driven development. More likely to have shifted left. More likely to automate a higher percentage of test cases. More likely to test upon each code change. How Do Commercial Tools Help? Commercial lab vendors help connect open source test frameworks to mobile and web platforms. Lab vendors facilitate a stable test automation and execution environment. This promotes greater test coverage for automation. Teams also encounter less “noise” that is associated with DIY solutions. Many commercial tools like Perfecto, also provide additional productivity features like test reporting, test creation, scalability, app debugging, and more. Related Reading: Buyer's Guide to Web & Mobile Test Automation ToolsRead eBookOpen source tools and automation frameworks are a necessary part of any automation project, but they do have their limitations.2021 Will Be the Year of Intelligent Test Automation 48% of those surveyed are looking to invest in automation solutions within the next year. Are you planning to invest in test automation solutions within the next year?Teams using commercial lab vendors are…. More likely to achieve continuous testing, integration, and delivery. More like to practice behavior-driven development. More likely to have shifted left. More likely to automate a higher percentage of test cases. More likely to test upon each code change. How Do Commercial Tools Help? Commercial lab vendors help connect open source test frameworks to mobile and web platforms. Lab vendors facilitate a stable test automation and execution environment. This promotes greater test coverage for automation. Teams also encounter less “noise” that is associated with DIY solutions. Many commercial tools like Perfecto, also provide additional productivity features like test reporting, test creation, scalability, app debugging, and more. Scoping Out Test Automation Tools? Download our Buyer's Guide to Web & Mobile Test Automation Tools.Read eBookGrowing interest in investment shows the increasing importance and prioritization of testing.A Year in Review & What’s to Come in 20212020 might have been the most challenging year for software testing and development to date. Almost overnight, teams had to find ways to develop, test, and innovate from remote — sometimes with less budget, and with different tools and processes. As we look to 2021 and beyond, some of the changes from 2020 will still remain a reality for teams as part of lessons learned, risk mitigation, and process improvements. Digital Transformation In 2020, organizations moved almost entirely to digital as their only way to engage with clients. This isn’t going away — it’s only made digital channels more dominant for business. With that change, organizations put greater emphasis on web and mobile technologies from an innovation and testing perspective. Serving global users through web and mobile channels requires rigorous testing, bigger coverage considerations, and continuous testing and monitoring of these services — for both functionality and performance (responsiveness and availability). In 2021, teams will increase their investment in mobile and web applications through progressive web apps (PWAs) and newer and more engaging mobile apps like AppClips/APKs to provide better user experiences and retain connections with customers. Here, cloud testing solutions will play a major role in serving as the unified testing solution or CFTA (continuous functional test automation). Scaled Test Automation With more testing done on digital platforms, teams show a greater increase in test automation investment. We have seen in 2020 not only test suite sizes growing for mobile and web apps, but also the adoption of additional technologies that can serve different personas within the teams. Frameworks such as Cypress and Flutter, together with more BDD adoption, have seen tremendous usage as developers and testers take more and more testing responsibilities. This is not going to change going forward as digital winners will continue with such investments. Additionally, the growing emphasis on digital testing will place more focus on accessibility test automation across platforms. This type of testing has become top of mind for digital businesses serving all types of users — with and without any disabilities — as well as to eliminate manual testing activities for accessibility that happen too late in the process. Agile Testing QuadrantsTeams Restructuring and Process With the emphasis on shift left and DevOps, and in light of 2020, we have seen a dramatic change in the way testing teams are structured. With over 42% of survey respondents identifying as QA practitioners residing within feature teams/squads, it is clear that the transition from COE into smaller and focused testing teams is happening across the board. As products become more complex, having testers in proximity to developers in smaller groups makes E2E software delivery much more efficient. In 2021 and beyond, we will still be seeing hybrid models of centralized testing teams that serve big organizations, but the trend will be to shift the testing practice into smaller, dedicated testing teams within feature teams. Feature TeamsEfficiency Opportunities To succeed in scaling test automation and ensuring digital quality, organizations who started looking into more intelligent testing processes and tools in 2020 will continue doing so in 2021. This spans test reporting and analysis abilities to cope with growing numbers of test executions, and more maintainable test creation through scriptless and self-healing tests. With the help of AI and ML within intelligent testing solutions, teams will be able to uplift their daily activities to obtain more value from testing, expedite their feedback loops, and deliver products with fewer escaped defects. About PerfectoPerfecto by Perforce enables exceptional digital experiences and helps you strengthen every interaction with a quality-first approach for web and mobile apps through a cloud-based test platform, comprised of real devices and real end-user conditions, giving you the truest test environment available. Our customers, including 50 percent of the Fortune 500 companies across banking, insurance, retail, telecommunications, and media rely on Perfecto to deliver optimal mobile app functionality and end-user experiences, ensuring their brand’s reputation, establishing loyal customers, and continually attracting new users. For more information about Perfecto, visit perfecto.io. Try Perfecto
