Open Source Mobile Test Automation Frameworks: How to Choose What’s Right For You
With that great statistic, it is also known that many of these organizations are struggling with the following:
- Maintaining the test automation suite
- Running the test unattended
- Dealingwith existing organizational skillset
- Extending test automation coverage
What we've learned is that not all open-source test automation frameworks are built the same, nor do they serve all organizational requirements.
As you can see from the dashboard above, when deciding between test automation frameworks, a few things need to be clear:
- Realize that for your specific need, one framework might not be enough
- There are tradeoffs that need to be acknowledged by the teams beforehand - see the No's above
- Decision making needs to be based on test scalability, as well as existingtool suites and IDE's
- Acknowledge your lab setup, environment, and IT requirements when picking your solution - at the end of day, it's your team that will need to maintain it
- Tests are aimed to run in an unattended fashion via a trigger, CI, or other pre-defined events throughout the SDLC - a test framework that is flaky or does not fit your goals, will interfere and delay your schedule
So, what is the golden path?
There are three pillars to guide you - Your lab, your automation requirements and the analysis post-execution. If you can get these all under one lab in a managed way, you're on the right path to success.
- Lab - At the heart of an unattended automation lies the lab that assures devices, browsers, and other digital platforms are always connected, available and in a ready state to execute your scripts. If the digital lab is also accessible from the IDE environments, that's even better.
- With a reliable lab, comes the following benefits:
- Flexibility as market shifts
- Autonomy across teams
- Complete E2E coverage
- Unattended testing - to sustain CI, and move faster
- Zero setup of environments
2. Automation - As mentioned above, not every automation framework can fulfill the entire set of requirements. In today's reality, multiple teams and personas are leveraging test automation for various goals; Dev builds validation, CI, quality goals and more. If the automation framework lacks certain capabilities, the entire project is at risk.
- If the automation framework that connects to the lab can support the various capabilities and personas, the end-result is greater test coverage and faster Dev-Build-Test cycles. Having an automation solution that is open and integrated enables flexibility and can match various skills across teams.
Some examples of common test automation requirements that are often a challenge within open-source tools are:
- Visual analysis
- Complete system level control
- Same day support for new OS versions
- Capabilities like network condition simulation, GPS etc.
3. Analysis - Having a lab and test automation framework that match the team's requirements is great, however, at the end of each test execution, the team needs an efficient way to drill down into failures, identify the root cause and deliver fast feedback to the developers to address and fix.
- That's why having a digital reporting capability is necessary for teams to take action and fix issues faster and earlier.
In order to choose the right open source frameworks and maximize value, teams need to assess their Lab at the core of their daily operation, the test automation capabilities and finally the reports that framework can generate - especially when it evaluates hundreds of test cases across multiple platforms.
- Selecting the Best Open Source Automated Testing Tool for You
- How to Scale Up Your Test Automation [eBook]
- XCUITest: The Emerging iOS Testing Framework for UI