View all web browser and mobile devices available in our cloud-based test lab.
A digital lab is a must-have for testing web and mobile applications. Here, we define what it is — and how to leverage it.
A digital lab provides access to real devices and browsers though an automation interface with a guaranteed level of uptime, or service availability, to support Agile development of web and mobile apps.
Mobile and web are now primary touch points for users. Thus, you need a digital lab capable of bringing the real user experience into the hands of developer and testers.
The Buyer's Guide to Web & Mobile Test Automation Tools
But there are complexity concerns and costs associated with building your lab, consisting of real devices and browser platforms. Here, we break down what it takes.
It's important to right-size the lab for your organization. To do so, we will reference the Test Coverage Index to identify the digital platforms to include in our lab as a representative set. You are encouraged to define your own devices and browsers based upon your customer data.
Device Farm vs. DIY vs. Testing Platform
From the index, we will select the top 25 devices and the recommended 33 browser/operating permutations. We assume the lab has the capability to test across all devices and browsers concurrently. The cost calculations are estimated using Perfecto’s lab calculator and summarized in the screen capture below.
To illustrate how the lab would scale based enterprise needs, we ran two scenarios through the tool, and summarized in the table below. The inputs are highlighted in yellow, and the summary calculations are displayed in black. We looked at these two scenarios:
It would cost over a half million dollars to build and staff a digital lab as a shared resource for a localized app team. Nearly half the cost is from manpower needed maintain the lab for continuous testing throughout regular working hours.
How to Select a Device Cloud, From a Customer Perspective
The automation service is assumed to use open source Appium and Selenium grids. The browser VMs and devices are connected to commodity servers (Windows/Linux) and Mac hardware.
To support a global team, the scenario would be expanded to at least three labs, to cover Americas, Europe, and Asia for example. Note that in this scenario, each lab operation has been expanded to 24x7 to support the global nature of the distributed team. This allows the teams to run nightly automated tests outside of the normal 8-hour workday in each region to support a global enterprise.
Cloud Testing Is More Critical Than Ever
In this scenario, the cost scales to over $4 million, almost 8 times the cost of a local lab, driven primarily in the labor to support 24x7 operation of the global labs. Note that each region would also have a different mix of devices and carriers that represent the region's demographics as well.
Looking at a cost of over $4M, you may be wondering about the cloud. Moving to the cloud would save you over 50% or more of this cost and would vary depending upon your specific needs.
If you are convinced the cloud is for you, check out our subscription plans to get started, or contact us to price a custom cloud hosted lab for your enterprise.
Try Testing With Perfecto
If you’re still considering building your own lab, continue reading through the next sections. To understand what the numbers are composed of and the assumptions behind them, let’s dive into the details. We’ll use the local app team as the reference to explore.
At a minimum, a digital lab requires seven components.
Devices under test are more expansive than you think. It costs around $6,200 per device to host a device in a lab. That is nearly 10X the cost of the device.
Why is it so expensive? It’s not as simple as connecting devices to a laptop. You need infrastructure and people to care and feed the devices, such as what’s shown in the image below.
A device lab consists of devices, servers, network, rack infrastructure (power, connection, Ethernet), health and safety certifications, and people to maintain the lab. The table below summarizes the expenses.
Note that the capital expenses are typically amortized over 3 years and reported in the Annualized Cost column as the actual yearly cost of the device lab. Year 1 total reports the cost if all capital was purchased outright in year 1.
Let’s drill down into each area.
Mobile devices are designed for consumer use, not for continuous automated testing under load. When placing a real device into a testing scenario, the average lifespan is around 9 months. Therefore, a lab of 25 devices would need to be replaced on average every nine months, or approximately 37 devices in total[i] throughout the year.
Each device is assumed to have a carrier contract at an average price of $55/month.
Hardware to access and control the devices consist of a mix of Mac and PC servers in the ratio of 1 Mac mini to 4 iOS devices, and 8 non-iOS devices per PC. This gives a requirement of 2 Macs and 3 PCs to support testing on 25 devices.
Infrastructure to support the devices consist of Wi-Fi access points, cellular repeaters (to support adequate cellular signal), and racks to host the devices and servers. One Wi-Fi access point can support up to 50 devices. An average cellular repeater can support 8 devices. The lab would require one Wi-Fi access point, and 4 cellular repeaters. One rack would be adequate to support 25 devices.
Care and feeding of the digital device lab and computers are required to maintain operation for automated testing. The devices require 3 hours/month/device on average of maintenance, diagnosing and fixing unresponsive devices, configuration issues, and so on. Similarly, Mac and PC servers need maintenance at a rate of around 2 hours/month/server.
And finally, if more than 20 devices are connected, a RF inspection to certify the lab has safe RF levels of operation needs to be completed. This is an annual cost of around $10,0000.
The net result is that it costs $6,200/device, or $156K annually for the full device service.
This is just to host all the devices as a service. We have yet to add into the ability to automate testing. We’ll get into that, but first let’s build out the desktop browser-as-a-service component of the lab.
A browser lab is just a collection of virtual machines, right? Yes and no. OS X complicates the standard model of spinning up a VM. Apple requires Mac hardware to host OS X. IT organization generally do not support Mac hardware for VM hosting or have the capability internally. So, Mac hardware needs to be introduced into the environment. A lab would look like the representation below.
Supporting browsers on OS X is more complicated, requiring Apple hardware and limitations on the number of OS X virtual machines that can be run off Apple hardware. In this case, we’ll assume using Mac minis, and OSX browsers are offered in a 1:1 ratio, to allow concurrent execution of testing on each machine. 8 Mac minis are required to support the OSX browsers requirements for the local app team lab.
For Windows and Linux based browsers, commercial VM software and servers can be used. For the analysis here, we assume one server will support 8 concurrent browsers.
The result is an average browser VM cost of $2,700, or $90K annual costs for the full browser service.
Setting up an automation environment is the final step to enabling the lab to for practical testing. The automation environment here will assume setting up open source Appium and Selenium grids with a storage area network (SAN) to save test logs, screenshots, or video recordings.
For the SAN, it is assumed that 3 TB of storage per year will be required for the 25 devices and 33 browsers under test, at an average cost of $6/GB. Installing and configuring Appium and Selenium is assumed to take 1 day of manpower each to implement.
The table below breaks down the cost for setting up and maintaining the Selenium grid, with annual re-installation of new releases assumed. Setting up the grid and maintaining it adds about $20K to the total.
The lab to this point costs around $260K per year. But it isn’t guaranteed to be running 8 hours x 5 days a week. There isn’t anyone monitoring the operation or troubleshooting issues.
At a minimum, the lab needs two people to monitor (Ops) and maintain (IT) the operation of the lab to provide the test team with assurance every device and browser will be available on-demand for digital testing during normal working hours.
This is a substantial increase, nearly 50%, bringing the total to $500K.
The key assumption here is that the lab must be always on during the work day to support an app team working in an Agile/DevOps enabled environment (i.e. testing in and outside of continuous integration processes). The reality is that the current assumption does not provide support outside of normal working hours, so nightly testing is not accounted for. This would add to the cost but was ignored for simplicity.
Of course, there's a lot more to test coverage than what's listed here. Learn more about test coverage in our test coverage index >>
You could build your own lab. Or you could take an easier way and use Perfecto's cloud-based lab today.
4 Ways Perfecto Delivers ROI
Check out Perfecto's smart cloud testing lab — complete with mobile devices and leading web browsers, AI-powered analytics, and streamlined test execution. But don't just take our word for it.
Try it for yourself. Get your free Perfecto trial now.