Developing a Top-Notch DevOps Testing Strategy

When thinking about DevOps and testing together we usually refer to continuous testing practices. The reason is, DevOps enforces fast releases of new value to customers that need to be validated quickly. Validating software quality quickly and continuously can only be done via smart test automation.

In this post, I’ll provide some fundamentals to building a DevOps testing strategy that can keep up with iterative software releases and provide high-value, data-driven feedback to the developers and managers.

DevOps Testing Strategy Factors

To build a valid test strategy, DevOps teams ought to acknowledge the varying operating forces within the squad or feature teams. In such teams, we would typically look at software developers, software test automation engineers (SDETs), and business analysts/testers. It’s important to match the strategy to these personas from both a technology perspective (i.e. test frameworks, tools), as well as from a process perspective.

An additional factor in building the right strategy is understanding the product’s architecture and domain.

Mobile apps, web apps, progressive apps, desktop apps, or others have different quality constraints and considerations. It can be around the tools, the coverage needs (mobile devices and tablets e.g.), the upcoming market events, and more. From an architecture perspective, if the product is designed and developed via micro-services as opposed to a monolith impacts the entire testing strategy.

An additional factor is the velocity and software release cadence, as well as the available resources allocated to the job. If the desired cadence is e.g. every two weeks as opposed to every 2 months, this obviously impacts the level of pressure on test automation, platforms, scalability, and many more.

Since continuous testing in DevOps relies on rigorous and robust testing infrastructure, things like clean-code guidelines, valid environments for dev, QA, staging, and production, and a standard process for continuous integration (CI) pipeline that serves the teams will be critical prerequisites for the future.

Lastly, teams cannot measure what they cannot see. Hence, data analytics and data visibility with actionable artifacts must be considered in the early stages so teams get the most relevant feedback as they move forward with the development.

In addition to the above factors, DevOps management needs to gather as many KPIs as possible from the teams such as user story development capacity, gaps in resources, skills, documentation, product-relevant history, and more to have a good starting point as well as level-set the expectations from the get-go.

DevOps Continuous Testing Strategy

After establishing the factors and gathering some important data points and KPIs, it is time to connect the dots into a solid strategy.

Here are the key pillars to building a top-notch strategy

1. Build the RIGHT test automation with the RIGHT persona.
2. Enable execution environments and labs that can serve them reliably.
3. Measure continuous success through agreed KPIs and improve.
4. Focus on test data analysis as individuals and as a team.

Build the Right Test Automation With the Right Persona

At this point when creating the test automation, teams must certify the test scenarios that they’re about to develop. Certifying means ensuring that the test case is valuable to be automated, that it covers the right objectives, and that it can perform well across platforms within CI and for the foreseeable future. When developing test automation, teams commit to maintaining the test until it either becomes obsolete and retired or requires a complete re-factoring due to changes in the product itself.

Developing the right test is great but developing by the right persona is a critical point. As mentioned earlier, the 3 personas that contribute to test automation share the load of test development, therefore, the distribution of the tests should match their technology and skills in such a way that they can “own” the entire lifecycle of the test independently, maintain, debug and monitor it as needed.

Enable Execution Environments & Labs

Having the right test automation scripts is great, however, having the proper environment that covers end-user conditions, and that includes the most relevant test or production data are crucial for success and test relevancy. Scaling the tests up and down based on product requirements, peak times and other constraints is an additional factor in the test strategy that teams rely on, which needs to be baked into the entire DevOps strategy. Often teams use outdated platforms, outdated test data, and are over-utilizing the platforms under test. These constraints are a huge blocker to DevOps productivity and proper planning with the right counterparts can help mitigate the issues earlier in the process.

Measure Success Continuously & Improve

Getting things right for the first time is almost impossible, and to know where you have room for improvement DevOps teams need visibility into KPIs that matter to the business. Defining proper KPIs around development productivity and quality and constantly reviewing them makes the difference between standing still and continuously growing the organization’s maturity.

A few KPIs that teams should investigate are:

• Percentage of manual testing activities and automation
• Build success rate per squad/feature teams
• Escaped defects outside of the cycle or to production with root-cause analysis
• Test automation effectiveness – Defect detection rates, test flakiness, etc.
• Test data and environments sync over time (are teams getting access to the most relevant test environments?)

On top of the above examples, teams can add additional KPIs that matter to the business and that can help the organization mature properly.

Focus on Test Data Analysis 

One of the key enablers for success is overall quality visibility. Management needs to know at any given time how the teams perform, how the apps look, how healthy the CI pipeline is, and many other insights. It’s clear that when implementing continuous testing in DevOps, the level of test data constantly grows, therefore knowing how to slice and dice the data and make sense out of the data is a critical aspect in the overall strategy. Understanding product areas that are buggier than others, platforms that are buggier than others, and test scenarios that are flakier than others can help in establishing continuous testing that really works for everyone. Without test data analytics and governance, the data only piles up and becomes a bottleneck in the entire software release cycle.

Strategic DevOps Testing Is Key to Success

Continuous testing in DevOps requires a well-defined process. Such a process relies on a culture that allows the various practitioners to collaborate and contribute equally to the testing efforts. With that in mind, providing these practitioners with the right test environments, test frameworks, and proper visibility is key to their success. Lastly, measuring success as an ongoing practice helps everyone understand their maturity level and where they can and should grow.

Explore Test Automation With Perfecto 

Experience how Perfecto can help you implement your DevOps testing strategy with our 14-day free trial.  

start trial