Selecting the Best Open Source Automated Testing Tool for You

There's a shift to open-source mobile test automation tools for testing happening today among developers and QA. And it's not just happening in mobile testing. Many mature technology sectors are adopting lightweight, vendor-transparent tools to fulfill the need for speed and integration.

As with many free and open-source software markets however, a plethora of tools complicates the selection process. How do you know what to spend time learning, integrating and deploying in your own environment?

This post aims to help you choose which open-source test automation framework to use based on a number of critical considerations. Keep reading to learn more about web and mobile automated testing tools. 

Criteria for Selecting an OSS Test Automation Framework

The rationale and advantages behind your choice of open-source test automation tools can be related to a few key benchmarks:

1. Ease of script development and execution (supports agile processes and short iterations)
2. Cross team collaboration capabilities (Both QA and Dev can easily use the same tools)
3. Match app platform with test development language (ObjectiveC/Swift for iOS, Java for Android)
4. No platform capabilities gap around testing (support for the latest OS features)
5. Support for real devices as well as emulators/simulators
6. Fully integrated tools within IDEs

Additionally, there are considerations that differ from mobile/web project to project:

What are the application use cases? What level of complexity needs to be tested?

• Heavy UI elements?
• Environment dependencies (Networks, GPS, Camera)?
• What OS versions and API levels should be supported?

Does the app support multiple platforms -- Mobile (iOS/Android)? Web?

Based on these considerations, you are far more likely to see long-term success with your automated testing efforts than if you simply dive in to a given framework without understanding the implications. You must choose your web and mobile test automation tools wisely.

So what are the most popular open source test automation frameworks?

When looking at today’s open-source mobile test automation landscape, there are five highly-adopted test frameworks:

• Selenium – The leading open-source test framework for web app test automation.
• Appium – Open-source test automation framework for mobile native, web and hybrid apps.
• Calabash – Behavior-driven development (BDD) test framework based on Ruby development language.**
• Espresso – Google open-source test automation framework within Android Studio.
• XCUITest – Apple's open-source test automation framework within XCode IDE.

Each of these frameworks are being sponsored by a different community and have unique benefits to their target platforms and respective audiences. Though general-purpose frameworks cover a broad range of devices, they often lack late-breaking hardware support. Frameworks that are device-specific often lack support for different scripting languages and approaches. It is therefore important to identify what is important to your team and project as part of the selection process, and avoid just selecting a framework based on technical requirements.

In the table below, we list the top web and mobile automation testing tools with their core capabilities and limitations:

Why Choose One Framework Over Another?

You may already have portions of test automation frameworks and tooling integrated into your software delivery process. These decisions may not always have been made by a single person or team, but rather a collection of experiences and motivations over time. A few of these motivations are:

• Teams trying to get fast quality feedback per each of their app builds and code commits
• Teams testing UI and functionality of their app
• Teams using behavior-driven testing tools to match their agile processes
• Performing cross-platform testing of mobile and web
• Complementing unit testing

The table above shows that not all web or mobile automation testing tools can provide full coverage of your requirements, and may even come with a risk of limiting the quality of the app under test do to their shortcomings. We have summarized each approach to help you decide which automation tools for testing make the most sense for you.

A Closer Look at the Top Open Source Automated Testing Tools

Appium

Appium is best suited to QA teams trying to test the functionality of native, mobile web and mobile hybrid apps across iOS and Android. This tool is less suitable for developers who wish to develop and perform unit testing since it uses a different scripting language than the app itself, (e.g. ObjectiveC). The generated Appium report is a bit limited from a debugging and fast feedback loop perspective, and does not include videos, network logs and key vitals information.

Selenium

The Selenium framework is the best choice for web test automation teams testing for RWD (responsive web design), or stand-alone web sites. It's less suitable for developing unit testing, which makes this framework less appealing for developers. Core Selenium test reports are not highly informative and lack unique mobile related insights.

Calabash

Calabash is designed for organizations that work in BDD (behavior-driven development) workflows. The tool offers an easy path to develop the features in parallel with the tests for these features in an easy user-flow based language. Calabash is appealing for both dev and QA practitioners. The tool provides solid insights and reports to Dev and QA teams.

Espresso & XCTestUI

These two tools are are very similar in that they are both designed for target users. Espresso is for Android and XCTestUI is for iOS. Both tools are fully integrated into development IDEs such as Android Studio/Xcode, and offer very easy to develop techniques, including test recorders. These tools are fully maintained by Google and Apple, which assures that they always support the latest OS features (i.e. iOS Force Touch) so developers can stay ahead of the market and test accordingly. These tools support both unit testing types and functional UI testing. Both tools are app context only, which limits their abilities to test for user condition scenarios.

Bottom Line

Not all open source automated testing tools are created equally. Some open source test automation tools are specific to platforms or operating systems. Some web and mobile test automation tools are sponsored by different communities. Some automation tools for testing are based on certain scripting languages. Some are device specific. But they all offer different benefits to users. Therefore, you must carefully select the best web or mobile automated testing tools for you.

**Note that Microsoft and Xamarin are discontinuing support for this framework and therefore any iOS versions greater than 11 and Android versions greater than 8 are no longer supported.