July 15, 2020

How to Select a Device Cloud, From a Customer Perspective

Automation

Continuous Testing

Looking for the right device cloud for your team? You’re in the right place. Hear how to approach this daunting task from a customer’s perspective.

One of Perfecto’s customers recently shared their experience of shopping around for a solution and the steps they took to carefully weigh their options. After all, there is a lot to consider when selecting a device cloud. And there are many solutions available on the market.

Keep reading to learn how they developed selection criteria, analyzed their options, and how you can do the same.

How to Select a Device Cloud

Cloud testing is critical for today’s testing teams. That’s why many teams turn towards a device farm. It offers the scalability and platform coverage needed in today’s fragmented market. But with so many options to choose from, where do you start?

Here are the steps our customer took to conduct a thorough investigation of all their options.

1. Align Objectives Internally

Many different types of solutions exist to meet the varied needs of all types of teams. So in order to find the right device cloud for your unique needs, you need to first flesh out exactly what your team needs. Aligning objectives internally up front will help you select the right device cloud.

Your team needs to understand...

Where a device cloud will fit into your processes.

What you expect of it and how it will help you.
How it can support your long term test strategy.
How a device farm can support your testing needs today and in the future.

By examining these objectives, your team can come to a better understanding of...

A high-level architecture of testing processes.
How the device farm will fit into existing tools and processes.
How the device farm will help you meet your testing goals.
Requirements you need from the device farm, including platform coverage, testing scope, and enterprise cloud integrations.

2. Research Device Cloud Options — Including DIY

Armed with internally-aligned objectives for a device cloud, the research phase of finding a solution will become much easier. Create scorecards for each device cloud you consider, ranking the most important criteria for your ideal device cloud solution.

Related Reading: Device Farm vs. DIY vs. Testing Platform

When examining device cloud solutions, don’t overlook doing it yourself. This helps you understand the components of a solution that are most critical to your team and internal limitations that may hinder your success.

One Perfecto customer first considered going the DIY route. They identified end-to-end test flows to try on their own automation framework. And while working with their development team helped them to overcome some of the automation limitations, they still faced a number of challenges by themselves. These included biometric identification, QR code scanning, the ability to send and receive SMS via cellular service, and use of front and back camera.

By identifying these limitations, the team better understood what to look for in a device cloud solution.

3. Give the Product a Try

Take advantage of free trials to try device clouds before you make your decision. With the free trial, see how your existing test automation frameworks work within the device cloud. Make sure that the device cloud can help you overcome the limitations you discovered in your DIY attempt.

Also be sure to see how well the device cloud integrates with your existing processes and toolchain. Get a sense of how the device cloud scales to your needs and your amount of testing.
Get a feel for the platform’s ease of use, and check to see if it meets security and compliance regulations. Lastly, get a sense of the cost of implementing the device cloud.

Based on how your free trials go, identify the top device clouds to carry out a proof of concept based on success criteria that aligns with your internal objectives. Again, you can apply the scorecard technique here to track all the important criteria for each device cloud. This can easily be summarized and shared with internal stakeholders when making your final selection.

Other Considerations When Selecting a Cloud Device Farm

Limitations of Free Trials

Keep in mind that free trials may not include access to every single feature in a solution. Keep an open dialogue with vendors about your needs and the features that are important to you. Even if such a feature is not available in the free trial, many vendors are more than happy to provide a custom demo of whatever features you would like to see.

Meeting the Needs of All Team Members

The reality of today’s testing is that there are fewer silos between teams. Quality is now considered the responsibility of everyone — no matter their skillset or objectives. Successful test automation can only be achieved with the help of the right tools. Support the needs of all team members. Look for test automation creation options that go beyond the basic scripting of test automation frameworks. Low code/no code options, such as BDD testing and codeless testing, can get more of your team involved with the testing process.

Full CI/CD Integration

A CI/CD pipeline allows teams to execute and analyze in minutes, understanding where the problems are in the code. It should come as not surprise that more teams are seeking to establish a CI/CD pipeline. When examining solutions, pay special consideration to the integration capabilities with CI/CD tools. A dashboard of all testing activities within the CI pipeline can be beneficial as well.

Real vs. Virtual Devices

When looking at device clouds, some may offer virtual devices. Some may offer real devices. And some may offer both.

All three types of testing have their own benefits and drawbacks. Emulators and simulators make testing apps on multiple mobile devices easier and faster. But they do not offer the test coverage of real devices. For that reason, you also need real devices. A combination of emulators, simulators, and real devices can provide teams with the optimal testing coverage.

Vendor Expertise/Support Services

When you’re examining device clouds, you should also examine the vendors themselves. Consider their expertise in the field — how much guidance or best practices can they offer for your testing strategy? Do they have a professional services team to help you along the way? You’ll want to ask about how they can get you up and running quickly. And, you should ask about their implementation plan.

Bottom Line

Selecting a device farm to partner with is no easy feat. It requires thorough investigation, the time to test your options, and clear internal dialogue to determine your objectives.

As you conduct your search, consider adding Perfecto to your list of device farms. Perfecto is more than a cloud — it’s an end-to-end testing platform for web and mobile apps. With 15 years of industry-leading experience, no one knows app testing like Perfecto.

Try Perfecto to…

Test against devices and browsers in the cloud.
Scale testing across platforms.
Integrate with anything in your toolchain.
Drill down on defects with test reporting and analytics.
And much more!

Start your free two-week trial today.

Try Perfecto

The 2023 State of Test Automation