How to Choose the Right Cross-Browser Testing Tool For You

The web landscape is evolving to meet the digital experiences and digital confidence that consumers expect. As a part of the evolution of the web, we see a growing adoption of PWA (progressive web apps) websites. We also see new versions of browsers with greater UX, such as Edge, which is based on Chromium. And the market is also experiencing a rapid cadence of browsers updates — Mozilla is releasing a new Firefox GA browser monthly.

To complement the rise of modern web technologies, cross-browser test automation frameworks are progressing and offering a wider than ever set of options. In this post, I would focus on the top four cross-browser testing tools that, while they seem to be competing with each other, are actually complementing each other.

Types of Cross-Browser Testing Tools

The market is flooded with cross- browser frameworks. We can divide them into four major categories: 

1. Open source frameworks that are derived from the Selenium WebDriver framework (WebDriver IO, Protractor, etc.).
2. Stand-alone frameworks (Cypress with JavaScript), Codecept, etc. 
3. Headless browsers testing tools (Puppeteer, PlayWright).
4. Codeless based (e.g. Selenium IDE, Perfecto Codeless).

At a high level, selecting from the above tools needs to be a matter of:

• Testing objectives — Functional, unit, exploratory, etc.
• Persona – Developer, SDET, Business Tester.
• Skillset – Coding skills, specific coding skills, non-coding skills.

In the below comparison, which is done in an arbitrary order, I will focus on the top four solutions.

Comparing 4 Cross-Browser Testing Tools

1.Introduction to Cypress

Cypress.io is a JavaScript based front-end test automation framework. 

Cypress is very intuitive and runs the tests within the browser, so the test creation, execution, and analysis are extremely fast, efficient, and developer friendly. Cypress is JavaScript and Mocha-based scripting tool. It is open source, but also comes with a commercial license for those who seek more users and more recordings per month (up to 20 users and 150,000 recording for the leading license).

Cypress Testing: Practical Considerations for Dev Teams >>

A sample script can be written in JavaScript and executed through the command line or the Cypress UI tool. Or, it can be recorded through the tool itself. A sample script and test execution looks like the below screenshot. The script is fully synchronized within the test runner, so clicking upon execution complete on the left command will show the DOM screenshot and the result.

Cypress is emerging as a dev-focused framework, with a great set of APIs, documentation, and examples. When looking at Cypress vs. Selenium, Cypress is limited in the number of browsers it can run/test against at this stage, and that’s a downside compared to Selenium.

Since Cypress has less mileage in the industry, it is more limited in its integrations and flexibilities to fit into other DevOps toolchains like Selenium might do. Later in the post, I will provide more comparison details. 

2. Highlights of Upcoming Selenium 4

Selenium is the leading cross-browser test automation framework. Selenium is a WebDriver-based solution. It is open source, which allows for the creation of test automation scripts in various languages including Java, JavaScript, C#, Python, Ruby, and Objective C.

Selenium technology together with its website has undergone a complete transformation over the past few Selenium releases. The upcoming Selenium 4 release will be 100% W3C compliant. And this will require changes in the scripts to make browsers compliant — starting with the desired capabilities definitions (e.g.: browserVersion instead of version), and more.

In addition, the new version will be packed with productivity features including relative locators, a modified Selenium IDE, a new Selenium Grid implementation, and integration with Chrome debugging protocol (CDP) for additional debugging abilities, together with other enhancements.

Selenium 4 Relative Locators Code Sample (Source: Software Test Academy)

Selenium, as opposed to Cypress, allows its users to create various types of testing including performance, security, unit, and others through its APIs. It also offers integrations and extensions to the framework itself.

What is common to both frameworks — including the headless ones — are the limited abilities of reporting and smart analysis, as well as parallel testing at scale. This is where integrations with cloud-based solutions like Perfecto complement these abilities.

Related Reading: 8 Benefits of Cloud Test Automation

3. Google Headless Puppeteer

Puppeteer is the modern headless browser testing for Chrome. This solution was developed by Google and has unique advantages and capabilities. It comes bundled with the latest Chrome browsers and allows users to run the tests within the browser. This provides fast feedback, debugging abilities, logs, and other chrome DevTools features that are accessible through the Puppeteer APIs.

This is a dev-friendly solution like Cypress and PlayWright and does not require setting up a complicated browser grid. It is also JavaScript based, which makes it very appealing to front-end web developers. 

Google Puppeteer Test Example (Source: GitHub)

Headless browsers by definition are serving specific requirements to the developers and test engineers. They can provide fast feedback and are typically great to use in the earlier development cycle of a website features. They also allow users to perform specific performance, security, and other audits like network and accessibility.

4. Microsoft PlayWright

Microsoft PlayWright is one of the newest and trending cross-browser testing tools. This framework was created by the same team that developed Puppeteer. One of the main differences between the two is the wider support for both Chrome, WebKit, and Firefox browsers within a single solution, as opposed to Chrome only by Puppeteer.

This is a node JavaScript-based framework that supports both Windows, Linux, and MacOS. From test runner support, it supports Jest, Mocha, and Jasmine as well as major CI servers like Jenkins, Azure Pipeline, Circle CI, and more. Among its rich features, PlayWright can support these additional capabilities:

• Scenarios that span multiple pages, domains, and iFrames.
• Auto-wait for elements to be ready before executing actions (like click, fill).
• Intercept network activity for stubbing and mocking network requests.
• Emulate mobile devices, geolocation, and permissions.
• Native input events for mouse and keyboard.
• Upload and download files.

Sample PlayWright JavaScript Test (Source: PlayWright.Dev)

Comparison Between the Cross-Browser Testing Tools

As highlighted in the above summaries, each of the four cross-browser frameworks are extremely powerful and rich with functionalities and benefits for both the developers and test engineers.

The selection criteria needs to be based on the different persona needs, skillsets, and phases within the web development cycle. In the below visual, we can learn about the great adoption, downloads, and active community that supports these frameworks. This is a great advantage and a good sign that these frameworks are being worked on, supported, and that they will continue to keep up with growing web technologies.

Download and Community Trends Comparison (Source: NPMTrends)

Bottom Line

The cross-browser testing landscape is undergoing massive changes, and there are more options for SDETs and developers to shift more testing left into the pipeline. Selecting between these four is not easy, since they are all great options.

To make the right decisions, the team must follow a rigorous process that is value-driven as well as quality driven. While three out of the above four are marked as dev friendly, this does not mean that they are not beneficial to SDETS as well.

However, in many cases, SDETs would have a larger scope for testing than developers. In those cases, Selenium might fit better. At the end of the day, the above four or a subset of them can complement each other towards a greater test automation coverage and a higher quality website application.

Cross-Browser Testing With Perfecto

Try pairing one of these four cross-browser frameworks with Perfecto. Perfecto can help ensure scalability, reliability, and advanced reporting, all from the cloud.

By automating cross-browser testing with Perfecto, you’ll maximize test coverage. You’ll get faster execution of tests across multiple browsers and devices. And you’ll get advanced reporting so you can accelerate feedback loops, debug, and ensure higher quality.

Ready to get started on cross-browser testing using Perfecto? Get started for free with a 14-day trial.

TRY PERFECTO

Related Content

• Selenium vs. Puppeteer: Which Is Better?
• Cross-Browser Testing With Selenium: Web Testing Tutorial
• How to Make a Cross-Platform Test Coverage Strategy
• Beginner's Guide to Test Automation Frameworks
• Cypress vs. Selenium: What’s the Right Cross-Browser Testing Solution for You?