April 23, 2020

Best Practices For a DevOps Test Automation Strategy

Continuous Testing

What is the role of test automation in agile and DevOps? DevOps is critical in the enterprise. And to be successful, you’ll need to have a good strategy for testing.

Back to top

What Is a DevOps Test Automation Strategy?

A DevOps testing strategy must be focused on continuous testing. Testing cannot be DevOps or Agile unless it’s continuous.

An important part of DevOps is continuous integration/continuous delivery (CI/CD). And between CI and CD, there should be continuous testing.

Without continuous testing, there will be:

  • Escaped defects.
  • Late delivery of software.
  • Unhappy customers.

📕 Related Resource: Learn more about What is DevOps?

Back to top

Benefits of a DevOps Testing Strategy

Here are some benefits you can expect from a strong strategy.

Deliver Faster Feedback

Deliver faster feedback. You don’t want your developers to have to go back to code they worked on 1–2 weeks ago to fix a bug. You want them to get faster feedback, so they can fix issues right away. And you can release faster. 

Cover More Platforms

Cover more platforms by implementing a strategy that scales test automation. This can be challenging, as mobile and web platforms are fragmented. But, when you utilize the right test automation tools, you can ensure coverage across more platforms in faster time.

Reduce Test Execution Time Constraints

Finally, reduce time constraints on test executions. Execution time needs to be shorter. When you run at scale, in parallel, and with the right tools, you can accelerate the time to market.

How do you ensure a solid strategy so you can experience the benefits? To do that, you need to follow these testing in DevOps best practices.

Back to top

Best Practices For Developing a DevOps Testing Strategy

The ideal state for most organizations is continuous testing. But organizations are struggling to find a solid foundation for test automation. Here are our best practices for developing a testing strategy.

Scale Test Automation

To build a successful strategy for DevOps testing, you’ll need to scale test automation — on web and mobile.

To Scale Web Testing

There are 3 ways organizations are scaling web testing today:

  • DIY (e.g., Selenium Grid in-house).
  • Containerized computing (e.g., using Docker and Selenium).
  • Cloud (e.g., Selenium Grid with Perfecto).

To Scale Mobile Testing

There are also 3 ways organizations are scaling mobile testing today:

There are two major solutions for scaling test automation for web and mobile (and across browsers). One option is to use open source frameworks (e.g., Cucumber). The other option is to do codeless (e.g., Selenium).

Automate All Testing Types

To have a successful strategy, you’ll need automation across all of your testing types. This includes functional tests as well as nonfunctional tests, including performance, accessibility, and unit code coverage.

Maintain A Stable Test Automation Suite

Automation within CI and outside CI is at the core of Agile and DevOps. Ensuring that automation continuously adds value — and meets the goals above — requires a stable test automation suite.

Your test automation suite must be continuously reviewed, audited, maintained, and refactored when needed.

As product code changes, so do testing. Make sure your teams are looking into the test scenarios and updating them whenever it’s required.

Use Supporting Frameworks

Another important step for your strategy is to use supporting frameworks.

Here are some examples of where to use supporting frameworks:

  • Provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber).
  • Provide assertion functions (Chai, Jasmine, Jest, Unexpected).
  • Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma).
  • Generate code coverage reports (Istanbul, Jest, Blanket).
  • Provide a browser or browser-like environment with control of scenario execution, UI testing, and more (Protractor, Nightwatch, Phantom, Casper, Selenium,, TestCafe).
  • Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble).

Rely on an Enterprise Class Lab

To release high quality fast, teams should rely on a robust lab:

  • That can scale.
  • That is secured.
  • That is always available for all types of testing.
  • That is up to date with all latest mobile and web platforms.

In addition, such lab should be able to support all of the advanced automation capabilities. This includes audio, location, image, and other digital related capabilities that teams need to automate.

Use Advanced Reporting and Analysis

At the heart of each testing activity there’s a test report. The more detailed and actionable the report is, the faster developers can figure out the root cause of failures and move forward.

Ensure that you’re using a robust reporting platform as part of your entire testing activities.

Use Headless Browsers

Headless browsers play a role in your testing strategy for web apps. These are developer-friendly tools. Using headless browsers helps developers deliver a more stable build to test engineers. Plus, headless browsers — like Google Puppeteer — are free.

Developers won’t need to set up Selenium Grid. They can just plug Puppeteer into their environment. From there, developers can do some performance auditing, accessibility testing, and some unit testing all in JavaScript.

This plays a role in scaling test automation, so you can succeed with your DevOps test strategy.

Include the Full Team in Testing

Your strategy for DevOps testing won’t succeed unless the full team is involved.

To do test automation at scale for each pull request and code change, you need everyone:

  • Business testers to do exploratory and structured manual testing — utilizing codeless test automation.
  • Software developer in testing (SDET) to create functional and non-functional code-based test automation scripts (via Selenium, Appium, etc.)
  • Software developer to create unit and build acceptance tests. 
Back to top

Testing Trailblazers: How Digital Winners Address Test Automation

What sets leading companies apart in the world of test automation? In this session, Johnny Lam, Solutions Engineer, draws on his experience working shoulder to shoulder with trailblazing global companies — from Fortune 500 to startups — to strategize, implement, and execute their test automation initiatives.  What have they done to speed up and scale their testing cost-effectively? Get the answers in this session.

Back to top

Tools You Need to Succeed With Your DevOps Testing Strategy

To succeed in your strategy for DevOps testing, you’ll need more than the best practices. You’ll need the right tools.

Here is a comparison table of some of the tools you’ll need:

Compare Tools For Your DevOps Testing Strategy
Selenium / Appium WebDriver 
Supported Dev LanguagesJava, C#, JavaScript, Python, Ruby, Objective C
Supported PlatformsChrome, Safari, Firefox, Edge, IE, iOS, Android
Supported Test FrameworksMocha JS, Jest, other superset on top of Selenium (Protractor, WebDriverIO, etc.)
Setup and ExecutionDownload relevant driver, set up a grid, network and location impacts execution speed
IntegrationsPlenty of integrations (CI/CD, reporting, visual testing, cloud vendors)
Breadth of Testing OptionsEnd-to-end, security, unit
Maturity, Documentation, SupportRobust community, multiple bindings, best practices
Required SkillsetCoding skills required (SDET Oriented)
Cloud and Execution at Scale

Perfecto fully supports Selenium and its WebDriver configurations

Local execution requires setting up a Selenium Grid

XCUITest / Espresso / Headless / Cypress 
Supported Dev LanguagesObjective C, Java, JavaScript
Supported PlatformsChrome, Electron
Supported Test FrameworksN/A
Setup and ExecutionEmbedded into IDEs, headless bundles a browser in the FW
Breadth of Testing OptionsUI, unit
Maturity, Documentation, SupportGood documentation and code samples
Required SkillsetDev Oriented
Cloud and Execution at Scale

Built-in Chrome/Firefox browsers in headless

Perfecto Cloud supports scaling Espresso/XCUITest

Supported Dev LanguagesIrrelevant, based on record and playback
Supported PlatformsAll
Supported Test FrameworksProprietary UI with underlying Selenium WebDriver APIs
Setup and ExecutionMostly SaaS/browser plugin installation
Breadth of Testing OptionsFunctional, UI
Maturity, Documentation, SupportGrowing, limited
Required SkillsetNo coding skills required (Business Tester Oriented)
Cloud and Execution at ScalePerfecto supports codeless in the cloud
Supported Dev LanguagesJava, Ruby, JavaScript, Kotlin
Supported PlatformsAll
Supported Test FrameworksJunit, Selenium, Appium
Setup and ExecutionOpen source, Maven/Gradle/TestNG
IntegrationsPlenty + APIs (e.g., rest assured)
Breadth of Testing OptionsFunctional
Maturity, Documentation, SupportRobust community, docs, adoption
Required SkillsetStep-definition development in code is required/scenarios are no-code (mix of Business Tester and SDETs)
Cloud and Execution at ScalePerfecto Quantum is a web/mobile BDD framework


Use DevOps Testing Tools With Perfecto

Perfecto integrates with the DevOps testing tools listed above. And Perfecto can help turn your DevOps testing strategy into reality.

With Perfecto, you can scale test automation for web and mobile. You’ll be able to test across browsers and get the best test coverage possible. And you’ll be able to utilize Perfecto’s cloud-based lab to automate even the most complex test scenarios.

Plus, Perfecto integrates with your DevOps toolchain, including CI/CD tools like Jenkins.

As a result, you’ll be able to:

  • Deliver faster feedback.
  • Cover more platforms.
  • Reduce time constraints for test executions.

See for yourself how Perfecto will help you build and execute on your testing strategy. Get started today with a free 14-day trial.

Try Perfecto


Related Content

Back to top