View all web browser and mobile devices available in our cloud-based test lab.
In this blog, we cover shift left testing, why it's not happening, and what you can do about it.
Shift left testing is the strategy of testing earlier in the cycle. It literally means shifting testing left to happen earlier in development.
And while it is lauded for its many benefits to DevOps teams, in reality, it has not been widely adopted. Why not?
To better understand the current state of automated testing and see the progress of shift left testing, Perfecto partnered with Gatepoint research to survey leading digital enterprises.
We also spoke with testing and DevOps experts to understand why shift left testing is not taking hold and what it will take to implement.
Keep reading to learn more.
Shift left testing offers a number of tangible benefits for teams.
Curious about the current state of shift left testing? Watch this panel webinar with QA industry oracles Michael Bolton, Eran Kinsbruner, and James Bach. It is a frank discussion on timely and timeless topics in the world of testing.
The benefits the shift left testing approach are clear. But our State of Test Automation report found that few teams are successfully implementing it.
Given the benefits of shift left testing, we wanted to know how testing is structured within organizations. For the vast majority of companies, testing is done by a mixture of both QA and dev. 81% of companies surveyed rely on QA for testing, either supporting dev or working within dev. Only 19% of companies surveyed rely on development teams for testing.
For teams that rely on development for testing, open source test automation frameworks and tools are most commonly used. And 56% are automating less than a quarter of their test cases.
On the other hand, companies relying on QA for testing are more likely to use commercial tools and automate more of their tests. However, these teams struggle with test instability/flakiness and false negatives. While these teams face skillset challenges for creating test scripts, they are investing in commercial tools to ensure their success.
Too many teams are waiting to test until after a new dev build is ready. Organizations need to test throughout the process rather than at the end.
The majority of companies surveyed test after a new development is ready or after a new build is ready. Only about 40% of companies test upon each code change or at the start of each software sprint.
To mature DevOps, most checkpoints must be checked automatically. The more human intervention there is, the slower the process will be. Automating such checkpoints does not mean these gates are being put at risk. Rather, it means teams found the right ways to automate the flow and approve it.
An example of a best practice can be something like this. Instead of test engineers waiting for a new build to start their test automation, developers can grant an early access build that has the right “tag” attached to it to allow test automation activities to begin.
The numbers tell us that shift left testing is not happening. But why? Hear five experts weigh in on their theories.
“In theory, shift-left makes total sense. Security bugs can be extremely costly when they make it to production,” says Kristin Davis of 42Crunch. Despite this, testing is not shifting left. Why? Davis offers two reasons: lack of alignment with leadership and current processes.
“If R&D is incentivized solely on delivering the functionality and meeting the deadlines, any additional steps and solutions that are not aligned with that will meet resistance. Of course everyone would agree that security is important, but if the team's sole mandate is to ship that business-critical app by the target launch date, everything else will be pushed to the side.”
“When R&D is not involved in tool selection, the chances are that the tools and processes may not fit them well. Developers can be very religious about their tools, whether it’s developer environments (IDEs) or continuous integration/continuous delivery (CI/CD) pipelines. If they are asked to use something ad-hoc that doesn’t integrate, they will view it as a total productivity killer.”
One reason testing is not shifting left is due to workload capacity.
“Teams need to free up resources to actually do the shift-left testing while still continuing with existing testing. If your resources are already seconded to projects and testing and then a new project wants them to do testing early, it doubles the workload,” says Neil Price-Jones, President of NVP Software Solutions.
Additionally, skepticism whether shift left testing holds true benefits could be slowing down its adoption.
“People are not convinced, myself included, that shifting testing left will cover all the cases that need to be done. I have yet to get a convincing argument that just doing this testing at the beginning is sufficient to remove the risk.”
Culture is another impediment to shift left testing.
“The biggest reason why testing isn't shifting left as expected is a mix of culture and fear. Usually when companies start moving toward that, there's too much testing on the part of the tester. And the reason for that is there are a lot of bugs affecting quality, and the developers are working furiously to fix them,” says Michael Fritzius, President of Arch DevOps.
“Shifting left takes some of the testing effort off the testers but places it on an already overworked dev team. The fear of changing how the work is done, and expecting that the style of work changes to account for it, is met with resistance.”
Another reason behind the lack of shift left testing could be the lack of testing represented in early phase discussions.
“The main reason shift left testing is not happening is product owners are not engaging the testers in the early phases of design and development. They do not give much importance to testing before the application hits production,” says Naveen Kumar Namachivayam, Performance Test Engineer at QA Insights.
“During the initial important project discussions, oftentimes testers are not invited. Especially, product owners never involve nonfunctional testers in their regular stand up meetings. They think test automation solves most of the testing effort. It is not possible to automate everything. If the testers are involved during the inception phase, shift-left testing delivers fruitful outputs.”
“One reason that shift left testing is not taking hold is that the scope of testing is not well defined,” says Perfecto Chief Evangelist Eran Kinsbruner. “If it is too large without proper justification, there’s no room to integrate it into the build cycle to shift left.”
For each build to be tested, Kinsbruner recommends the scope of the test suites includes:
It’s also important to ensure your test suites remain valuable over time. Teams need to audit and maintain their suites between builds. This ensures that the testing scope is relevant over time.
Shift left means shifting testing left in the cycle, which allows you to test earlier. Testing earlier means bugs are less costly and time-consuming to fix. For these reasons, shift left testing is encouraged as a best practice.
Shift right in DevOps is about moving testing to later in the cycle. Moving testing into production allows you to see the end user experience and performance of the app
A few things still need to happen for the shift left approach to take hold. Shifting left needs to align with leadership’s vision. Teams need to be ready to show tangible results. And testing must be represented in the earliest phases of the project.
Learn more about current testing trends in the community. For more information on the current state of testing, download our report, The 2020 State of Test Automation.
Or, try testing with Perfecto today. Perfecto allows you to…
Start your free trial today.
VP of Product Management, Perfecto
Tzvika Shahaf is the VP of Product Management at Perfecto. His experience includes business development, strategy, and investment in technology companies and venture capital firms. His passion is building new, powerful, and effective ways to collaborate with Global 2000 enterprises in order to resolve high-impact business problems using data-driven processes and analytics. Tzvika is partnering with leading DevOps teams to revolutionize the testing space by making it smarter, faster, and cost effective with a clear goal of maturing software delivery lifecycle. Tzvika is keynote speaker at industry leading events, blogger, and a Co-Author of the book, “Continuous Testing for DevOps Professionals: A Practical Guide from Industry Experts.”