View all web browser and mobile devices available in our cloud-based test lab.
The web landscape is evolving to meet the digital experiences and digital confidence that consumers expect. As a part of the evolution of the web, we see a growing adoption of PWA (progressive web apps) websites. We also see new versions of browsers with greater UX, such as Edge, which is based on Chromium. And the market is also experiencing a rapid cadence of browsers updates — Mozilla is releasing a new Firefox GA browser monthly.
To complement the rise of modern web technologies, cross-browser test automation frameworks are progressing and offering a wider than ever set of options. In this post, I would focus on the top four cross-browser testing tools that, while they seem to be competing with each other, are actually complementing each other.
The market is flooded with cross- browser frameworks. We can divide them into four major categories:
At a high level, selecting from the above tools needs to be a matter of:
In the below comparison, which is done in an arbitrary order, I will focus on the top four solutions.
Cypress Testing: Practical Considerations for Dev Teams >>
Cypress is emerging as a dev-focused framework, with a great set of APIs, documentation, and examples. When looking at Cypress vs. Selenium, Cypress is limited in the number of browsers it can run/test against at this stage, and that’s a downside compared to Selenium.
Since Cypress has less mileage in the industry, it is more limited in its integrations and flexibilities to fit into other DevOps toolchains like Selenium might do. Later in the post, I will provide more comparison details.
Selenium technology together with its website has undergone a complete transformation over the past few Selenium releases. The upcoming Selenium 4 release will be 100% W3C compliant. And this will require changes in the scripts to make browsers compliant — starting with the desired capabilities definitions (e.g.: browserVersion instead of version), and more.
In addition, the new version will be packed with productivity features including relative locators, a modified Selenium IDE, a new Selenium Grid implementation, and integration with Chrome debugging protocol (CDP) for additional debugging abilities, together with other enhancements.
Selenium 4 Relative Locators Code Sample (Source: Software Test Academy)
Selenium, as opposed to Cypress, allows its users to create various types of testing including performance, security, unit, and others through its APIs. It also offers integrations and extensions to the framework itself.
What is common to both frameworks — including the headless ones — are the limited abilities of reporting and smart analysis, as well as parallel testing at scale. This is where integrations with cloud-based solutions like Perfecto complement these abilities.
Related Reading: 8 Benefits of Cloud Test Automation
Puppeteer is the modern headless browser testing for Chrome. This solution was developed by Google and has unique advantages and capabilities. It comes bundled with the latest Chrome browsers and allows users to run the tests within the browser. This provides fast feedback, debugging abilities, logs, and other chrome DevTools features that are accessible through the Puppeteer APIs.
Google Puppeteer Test Example (Source: GitHub)
Headless browsers by definition are serving specific requirements to the developers and test engineers. They can provide fast feedback and are typically great to use in the earlier development cycle of a website features. They also allow users to perform specific performance, security, and other audits like network and accessibility.
Microsoft PlayWright is one of the newest and trending cross-browser testing tools. This framework was created by the same team that developed Puppeteer. One of the main differences between the two is the wider support for both Chrome, WebKit, and Firefox browsers within a single solution, as opposed to Chrome only by Puppeteer.
As highlighted in the above summaries, each of the four cross-browser frameworks are extremely powerful and rich with functionalities and benefits for both the developers and test engineers.
The selection criteria needs to be based on the different persona needs, skillsets, and phases within the web development cycle. In the below visual, we can learn about the great adoption, downloads, and active community that supports these frameworks. This is a great advantage and a good sign that these frameworks are being worked on, supported, and that they will continue to keep up with growing web technologies.
Download and Community Trends Comparison (Source: NPMTrends)
Selenium – SDET Friendly
Chrome, Safari, Firefox, Edge, IE
Supported Dev. Language
Java, C#, Java Script, Python, Ruby, Objective-C
Test Runner Frameworks
Mocha JS, Jest, other super set on top of Selenium (Protractor, WebDriverIO, etc.)
Local grid, cloud
E2E, Functional, Unit, Performance
High, rich community and documentation, integrates with majority of CI/CD tool stack
Cypress.io – Front-End Dev Friendly
Chrome, Electron, Firefox, Edge
Dedicated UI, built-In browser, no grid
Front-End, Unit, UI
Growing, good documentation, limited integrations, limited use cases (multi-tab, hover, etc.)
Puppeteer – Front-End Dev Friendly
Node library installation, built-in browser, no grid
Unit, Security, Network, Performance, Accessibility
Growing, good documentation, fewer integrations than PlayWright
PlayWright – Front-End Dev Friendly
Chrome, Firefox, WebKit (Safari)
Mocha, Jest, Jasmine
Growing, good documentation, more integrations than Puppeteer
The cross-browser testing landscape is undergoing massive changes, and there are more options for SDETs and developers to shift more testing left into the pipeline. Selecting between these four is not easy, since they are all great options.
To make the right decisions, the team must follow a rigorous process that is value-driven as well as quality driven. While three out of the above four are marked as dev friendly, this does not mean that they are not beneficial to SDETS as well.
However, in many cases, SDETs would have a larger scope for testing than developers. In those cases, Selenium might fit better. At the end of the day, the above four or a subset of them can complement each other towards a greater test automation coverage and a higher quality website application.
Try pairing one of these four cross-browser frameworks with Perfecto. Perfecto can help ensure scalability, reliability, and advanced reporting, all from the cloud.
By automating cross-browser testing with Perfecto, you’ll maximize test coverage. You’ll get faster execution of tests across multiple browsers and devices. And you’ll get advanced reporting so you can accelerate feedback loops, debug, and ensure higher quality.
Ready to get started on cross-browser testing using Perfecto? Get started for free with a 14-day trial.
DevOps Chief Evangelist & Sr. Director at Perforce Software, Perfecto
Eran Kinsbruner is a person overflowing with ideas and inspiration, beyond that, he makes them happen. He is a best-selling author, continuous-testing and DevOps thought-leader, patent-holding inventor (test exclusion automated mechanisms for mobile J2ME testing), international speaker, and blogger.
With a background of over 20 years of experience in development and testing, Eran empowers clients to create products that their customers love, igniting real results for their companies.