Test Automation in a DevOps World

While companies have benefited tremendously from implementing DevOps processes, tools, and general best practices, certain teams within these companies have stood to gain more from DevOps than others. One team, in particular, that has needed to shift their way of working tremendously since the onset of DevOps is the QA team, which is often made up of manual and business testers. Testers continue to struggle with the need to produce quality work while adhering to strict deadlines imposed on them by development teams and managers. This pressure to continuously speed up performance while maintaining their same level of work has had testers looking for different solutions to address this issue.

This eBook will discuss the changing role of the software tester in a DevOps world, and the impact test automation has had in helping testers adapt and thrive in this DevOps era. This eBook will also include different suggestions for how testers can develop an effective DevOps testing strategy so they can best determine ROI from their testing efforts.

Background: The evolution from waterfall to agile

Before we dive into how testers and test automation need to function in the world of DevOps, we need to first understand how these concerns even came about in the first place:

The waterfall model of software development

In the previous century, before the term “agile” became mainstream, many companies adopted a waterfall method of software development. In a waterfall environment, developers and QA were two separate teams and operations running parallel, or more precisely, one after the other. While there are plenty of companies that still operate using waterfall today, this was especially popular in earlier years due to the constrained technology for the masses that was available at the time.

The waterfall paradigm has two main advantages: both the developers and QA teams are experts in their respective domains. While their worlds may be narrow, they rule them with conviction and excellence. The other advantage is the ease and strength of maintenance. Each team takes special care to modify, update and maintain its part of the equation.

Yet, there are clear disadvantages to waterfall that eventually drove to innovation of the entire development process:

QA testers have limited knowledge of the product being developed, preventing them from testing their applications properly or thoroughly
The process is long, since it consists of two cycles that don’t overlap; QA can only begin after only after the developers complete a development run
It is expensive. Companies take a long time to churn out new updates and invest heavily in maintenance, while needing to have two completely separate teams.

Many companies found that these disadvantages ultimately led to a lesser quality product, and started looking for a new way to develop their applications that better utilized their resources.

The transition to agile and DevOps

As technology became more and more ingrained into everyone’s daily lives, the need for faster releases paved the way for agile software development to increase in popularity. The agile model allows for faster software updates by breaking the product or application down into small, incremental builds. Creating many iterations during application development allows for teams to focus on adapting the product and taking customer satisfaction into account before the application is fully ready.

The many iterations that come with agile development also paves the way for more collaboration, as teams need feedback on a more frequent basis in order to move on successfully to the next stage. On the surface, moving to agile seems to solve many of the pain points that come with waterfall; both QA and development are able to control their respective domains, they can do so in a more cooperative manner that ultimately lets companies release updates to their user base more quickly.

But many QA teams find that while Selenium is open-source, it has other hidden “costs” when it comes to actually using the tool. Selenium is difficult to set up and maintain without coding knowledge. Plus, it creates a huge learning curve for testers to gain the skills required to work inside Selenium effectively. For many companies, the time and resources it takes to make the investment in Selenium worthwhile make it a less than viable option.

This push towards agile work also led to more companies striving to implement DevOps best practices. By automating the processes between software development and IT teams, DevOps gives companies the opportunity to build, test, and release software faster and more reliably. When implemented successfully, DevOps helps companies undergo a major cultural shift that helps them become more efficient, productive, and innovative.

Source: agilest.org

Yet, while certain teams benefited greatly from agile and DevOps, others still found themselves facing many hurdles:

DevOps and the software testing bottleneck

This shift to agile led to a larger focus on developers and DevOps processes. This means that developers often found themselves with the most up-to-date tools and methods, since the goal was to release the most cutting edge technology as quickly as possible.

On the other hand, manual testers and other QA professionals often had to work with legacy tools and couldn’t move quickly enough to meet company needs. This transformed testing into a bottleneck area, where QA teams were struggling to increase their testing speed while maintaining their high quality of work.

For many QA professionals, adopting Selenium was a great solution for testers to keep up with agile and DevOps processes. As Selenium is the most common open-source code with a steadily growing user base worldwide, this seemed like a great way for testers to come of age in an agile era.

But many QA teams find that Selenium presents its own set of challenges that still make software testing a DevOps bottleneck. Selenium is difficult to set up and maintain without coding knowledge. Being such an exhaustingly technical solution, Selenium creates a huge learning curve for testers to gain the skills required to use it effectively.

In addition, the highly technical nature of Selenium makes test maintenance difficult in an environment that requires frequent modifications and additions of features. These constant changes make many Selenium tests extremely brittle, since every small change in the application will cause the test to break. Testers then need to focus their time on maintaining the code in every instance where the affected area of the application is part of the test. This is why, for many companies, the time and resources it takes to make the investment in Selenium worthwhile make it a less than viable option.

The role of test automation in a DevOps environment

Within a DevOps environment, leveraging a sustainable and scalable test automation solution is more important than ever. Even in organizations where QA and DevOps do not work as closely together, any QA bottlenecks will undoubtedly have a similar effect on development. When automated tests are flaky, companies will need to deal with issues such as false positives, inconsistent test results, and manual executions of important, yet complex test cases. These issues will inevitably lead to delayed releases, preventing developers from creating new features and capabilities that will move the application forward.

The impact of codeless test automation on DevOps

Codeless Selenium test automation addresses many of these challenges as companies try to progress in their DevOps journey. First, codeless test automation allows QA professionals to create resilient and maintainable tests while leveraging the capabilities of Selenium, all without needing to write code. This paves the way for QA professionals to keep up with shorter release cycles by producing quality test flows at scale.

Second, a codeless test automation platform provides your entire test automation framework built-in from the get-go. This includes the ability to integrate with many other tools are essential to achieving a successful DevOps pipeline. TestCraft, in particular, integrates with popular CI/CD tools such as Jenkins, TeamCity, BitBucket, and others. This will make adding test automation to your DevOps pipeline a seamless and simple process.

Example of TestCraft’s scheduling capabilities through different CI/CD integrations

Third, codeless test automation can provide even more value when it leverages machine learning to reduce the time testers spend on test maintenance. When dealing with dynamic content, the right codeless test automation tool can help identify the right element and ensure that the test continues to execute properly. This will free up time and resources for both testers and developers to focus on more complex tasks while also steadily building up their automation testing efforts.

While one way testers can benefit from machine learning is by reducing test maintenance efforts, there are many other ways that testers can also leverage machine learning as well. Some of these include results analysis, providing insight into future data set analysis, and showing issues within the test scripts’ logic.

Regardless of how they utilize machine learning, automated testing tools that incorporate this technology into their platforms will continue to increase the positive impact made on the DevOps pipeline. These tools will help test suites become more reliable and resilient, creating a strong foundation for integrating them more thoroughly into the company’s development cycle. Popular benefits of making testing a more integral part of your DevOps environment include:

Finding bugs earlier in the development process (with all the time and cost-savings that entails)
Optimizing your testers’ and developers’ workloads with an increased focus on developing new features and exploratory testing
Creating a happier workforce by reducing their repetitive, monotonous work

Creating a DevOps testing strategy

Now that we understand how testing can create a stronger DevOps environment, how should companies go about putting this into action?

When thinking about DevOps and testing together we usually refer to continuous testing practices. DevOps enforces frequent releases to customers that need fast and efficient validation, and validating the quality of software on a continuous basis can only happen via smart and strategic test automation.

Here are some fundamentals to building a DevOps testing strategy that can keep up with iterative software releases and provide high-value, data-driven feedback to both developers and managers.

DevOps testing strategy factors

To build a valid testing strategy, DevOps teams need to acknowledge the various individuals that can own the testing process. In such teams, we would typically look at software developers, software test automation engineers (SDETs), and business testers. It’s important to match the strategy to these personas from both a technology perspective (i.e. test frameworks, tools), as well as from a process perspective.

An additional factor in building the right strategy is understanding the product’s architecture and domain.

All apps, whether mobile, web, progressive, or desktop, have different quality constraints and considerations. These can span a wide range of factors such as tools, coverage needs, and different market trends. From an architecture perspective, one would approach testing a product differently when it’s designed and developed via micro-services as opposed to a monolith.

The velocity and software release cadence also contribute to building a testing strategy, as well as allocating available resources to the job. For example, companies that want their release cycles to be every two weeks instead of every two months will consequently put more pressure on test automation, platforms, scalability, and other components.

Since continuous testing in DevOps relies on a rigorous and robust testing infrastructure, other critical prerequisites to forming a comprehensive test automation strategy include:

Clean-code guidelines
Valid environments for developers
QA
Staging and production
A standard continuous integration (CI) process

Lastly, a DevOps testing strategy needs to consider the data analytics and data visibility with actionable artifacts in the early stages. This way, teams can get quick and relevant feedback as they move forward with development.

Consider the testing persona in your test automation strategy

After establishing these factors, you are ready to create your test automation strategy.

The first thing to do when building a testing strategy is to allocate the right tests to the right testing persona. Whether it’s a developer, business tester, or SDET, the tests that fall under their purview should match their respective technological and business skill sets. This will help each individual take ownership of the entire test’s lifecycle independently, as well as monitor, debug, and maintain it as needed.

At this stage of test automation, teams must certify the test scenarios that they’re about to develop, meaning that they should ensure that it’s actually beneficial to automate a particular test case. Certifying the test also ensures that it covers the right objectives and that it can perform well across platforms and within the CI for the foreseeable future. Developing test automation for specific scenarios requires the team to commit to maintaining the test until it becomes obsolete and unnecessary, or it requires a complete makeover due to changes in the product itself.

Leverage the right execution environments and labs

While having the right test automation script is useful, having the proper environment that covers end-user conditions, including relevant test or production data is absolutely essential for the success of the test and its relevance. Scaling the tests based on product requirements including peak times is an additional factor that needs to be incorporated within a DevOps strategy. Often teams use outdated platforms, test data, and over-utilize their testing platform. These constraints pose a huge obstacle to DevOps productivity. Proper planning with the right counterparts can help mitigate such issues early in the process.

A renewed focus on soft skills for testers

In addition to using automation testing tools and creating an airtight strategy to best adjust to the demands of a DevOps environment, testers will also find themselves increasingly focused on developing their soft skills when adapting to the DevOps era. Especially due to the major emphasis on culture when creating an agile and DevOps environment, software testing increasingly requires a healthy mix of technical and business savvy.

A 2018 study has shown 64% of job advertisements for software testers ask for soft skills. This chart shows a breakdown of skills requested.
Source: Software Tester, We Want to Hire You! an Analysis of the Demand for Soft Skills

There are four major areas where testers can focus on growth from a soft skills perspective include: asking the right questions, negotiation skills, product knowledge, and effective communication.

Asking the right questions

When doing software testing in a DevOps environment, critical thinking is especially important. While automated testing tools can minimize a lot of effort that manual testers often put into repetitive and menial tasks, much of software testing is context-driven, which only a tester who can ask the right question at the right time can manage. This is especially true when understanding how bugs might affect different features or capabilities of a product; there are a lot of unknown factors that tools can help with, but only testers can solve.

Another important element of asking the right questions is knowing when to prioritize them. Especially in an agile company where things are changing and updating constantly, testers are always racing against the clock to test as widely and as in-depth as possible before releasing software into production. Testers will ultimately thrive when they can determine when to delve into more granular components of an issue and when to keep questions open-ended This will help test automation go more smoothly and help testers better develop a strategy that will expand their test automation efforts over time.

Negotiation skills

Also known as “making friends and influencing people,” negotiation skills is something that testers should not overlook. It’s often the case that testers find themselves needing to make different business cases without actually having the decision-making authority to do so. In a case like this, it’s critical for testers to know how to negotiate and align the QA team’s needs with the larger needs of the company. Not only will this have a larger positive effect on the company, but it will also ensure that the QA team has the resources they need to do its job well.

There are many areas where QA will need to tap into this ability to influence without authority, from hiring personnel to convincing management to invest in different tools that the team needs (such as an automated testing tool). As already mentioned, the best way to approach situations like this is by focusing on how this request can help impact the company’s larger business goals. Whether this is by explaining future company benefits or showing concrete ROI, testers must focus on these larger points to show that what you’re asking for is worthwhile.

Knowing your product

While this is already a natural priority for any QA tester, the ability to learn the ins and outs of the application under test quickly and thoroughly is another important skill that will help testers succeed in a DevOps environment. It’s essential to remember that any application doesn’t just exist in a vacuum. Rather, the development team is building and updating this application in order to create a seamless user experience for a specific target audience, as well as address a variety of needs that your customers might have.

When getting to know the product, it is important to understand the various real-time business situations that may arise for end-users in addition to the product’s technical capabilities. This effort comes with many different benefits. Gaining deeper insight into how people interact with the application will help the QA team understand the application’s boundaries, and confirm whether or not newly added processes will continue to produce positive results for the company’s audience. This will help the QA team prioritize which tests are most important, plus inform future exploratory tests with this in-depth level of insight.

Another reason why knowing the product is an important soft skill for QA testers is that it will inform the feedback that QA team leads or the testers themselves will give the development team. Which leads to the next important skill that testers need to succeed in the industry:

Communicating effectively

One of the biggest challenges that companies trying to implement DevOps best practices faces is a lack of effective communication between developers and testers. Aside from the difference in technical skills, there is also a difference in priorities. While developers typically strive to create the most efficient code possible, testers often are more concerned that the application meets the relevant user needs while functioning properly.

A great way to improve communication is by encouraging each team to learn about the other’s experiences, areas of strength, and goals for learning. A lot of good communication stems from strong relationship-building, and learning more about people on different teams is a great way to start. Getting a better sense of the people you work with will ultimately help pave the way for stronger collaboration between testers and developers. Not only will both teams be able to interact with each other more empathetically, but they will also learn who is best to work with for different tasks. This will help create a more positive and open work environment, which will enhance the team’s ability to produce higher quality work.

Measuring the ROI of testing in DevOps

This eBook has discussed how test automation can help software testers adapt to a DevOps environment, as well as other skills and strategies testers can work on to address these changing technological shifts.

Yet the way to really show the company at large that investing in automated testing is worthwhile is by taking concrete steps to measure the ROI of DevOps testing. Getting things right on the first try is almost impossible. Figuring out where there is room for growth is even harder. However, testers and other teams that are involved in DevOps can identify areas for growth by creating the right KPIs. Defining and reviewing proper KPIs around development, productivity, and quality, makes all of the difference between keeping an organization at a stand still and promoting continuous growth.

Some KPIs that testers can start to focus on include:

Percentage of manual testing activities vs. automation
Build success rate per squad/feature teams
Number of defects that escaped outside of the cycle or to production with root-cause analysis
Test automation effectiveness – Defect detection rates, test flakiness, etc.

In addition to these ROI points mentioned above, teams can add additional metrics to track that can help the organization mature properly and correlate directly with larger business goals.

Focus on test data analysis as individuals and as a team

In order for testing in DevOps to be truly successful, management needs to have overall quality visibility and know at any given time how the QA team and these initiatives are performing. It’s clear that when testing continuously in a DevOps environment, the level of test data constantly grows. Therefore knowing how to parse and make sense out of the data is a critical aspect in carrying out a successful DevOps testing strategy. Understanding product areas that are especially buggy and test scenarios that are especially prone to breaking can help in establishing automated testing best practices that work for everyone. Without test data analytics and governance, the data only piles up and becomes a bottleneck in the entire software release cycle.

Test automation in DevOps requires a well-defined process, which relies on a culture that allows the various practitioners involved to collaborate and contribute equally. With that in mind, providing everyone from testers to developers and product managers with the right testing tools, environments, frameworks, and proper visibility is key to their success. When implementing these efforts and measuring their successes as an ongoing practice, everyone involved can better understand the company’s maturity level and where they can and should grow.

Mobile & Web Test Coverage Guide

Table of Contents: