July 12, 2019

5 Things to Consider When We Talk About Test Coverage

Automation

Mobile Application Testing

Today’s technology consumers have high standards. They expect their apps to run smoothly without bugs interrupting their digital habits. In fact, when Instagram is down, the outage makes national news. With the growth of IoT powering important devices like home security systems, errors and outages can have serious consequences.

Technology is not just expanding, it’s also fragmenting. Desktop browsers like Chrome and Firefox release a public beta or GA every month. And that’s just browsers. New mobile devices, versions, and operating systems are released daily.

With each release, a new digital testing challenge is introduced for DevOps. Test teams are expected to keep up with every advancement, test against every change, and identify regressions.

Test Automation Strategy For Beginners >>

To meet these steep customer expectations, testing teams need to be strategic. It’s essential for DevOps to have robust testing coverage that proactively analyzes the market and the pace of tech. Like test teams and developers retire irrelevant and aged code and test scenarios, same goes with platforms and OS versions — these need to be monitored continuously and adjusted as needed. Ignoring the importance of test coverage is the biggest risk of all. It puts your brand reputation at risk and creates an opening for your competition to win.

Now that we’ve emphasized why agile test coverage is so important, here is exactly how to create a bulletproof test coverage plan.

Establish a Full Picture for Test Coverage

The most important aspect of test coverage is understanding of the full picture from browser usage to devices. Test coverage is not as simple as it once was. With today’s rapidly-evolving technology spectrum, test coverage must be wide and strategic.

The 2021 State of Test Automation >>

The traditional definition of test coverage is often limited to what devices to test. However, the strongest test coverage includes a much broader list of considerations. For example, even platforms vary in their supported technologies. Some devices support face recognition, while others of the same platform family do not. So, it’s vital to include the right mix of devices and OS versions to cover all the required functionalities of your app.

Optimal testing covers:

Mobile device types
Web browsers
Operating systems
Multiple test types
Efficient and varied test timing and duration
Attention to the overall execution window

Strategic Test Coverage Includes:

1. Prioritizing Devices and Browsers for Testing

While new mobile devices and browser upgrades are appealing to consumers, constant updates can be a headache for testing teams. With every iteration, you must factor in the device, the screen characteristics, the OS, and the OS version. As you can imagine, the variations are nearly infinite.

Mobile App Testing Strategy Combining Virtual and Real Devices >>

The best test coverage cannot possibly test every device and browser combination. Instead, the test coverage plan must be a prioritized plan. A prioritized test coverage plan is created through multiple parameters.

2. Device Market Share and Demographics

The most popular devices in your demographic must be top priority for testing. So, it’s important to know what your demographic is, and what they’re using. It’s not enough to know the most popular browsers and phones. Your demographic data needs to be specific to the region or country you’re targeting. Obviously if your end users are distributed across different geographies, you should factor this into your testing lab setup.

3. Most Popular Devices and Browsers

It’s also crucial to keep tabs on the top device brands like Motorola, Samsung, Huawei, Xiaomi and LG. Furthermore, screen size and resolution have become a critical testing factor as devices are both getting smaller and larger to meet the particular preferences of the market.

3 Benefits of Testing on Dedicated Devices >>

4. OS Popularity and Adoption

Device type is just one factor of test coverage. The OS being used is also a paramount piece of the test coverage puzzle. OS frameworks can vary widely from one another so testing can’t be limited to one operating system like iOS.

The Android family of operating systems has the lion’s share of the market, but within Android are an incredible number of variations which adds major testing complexity. With iOS, while the reach is smaller, most users are updated to the latest OS.

5. Mimicking the User’s Environment

Ultimately, the only way to absolutely ensure a quality user experience is to mimic your user's environment. This means to replicate as closely as possible your user’s journey as they interact with your app.

Real User Simulation Testing >>

To accurately reflect your various user personas, DevOps teams should use the following tools and considerations on top of the unique device/OS models that were picked based on the above-mentioned considerations:

Analytics
App user stories
End-user profiles
Persona (Male, Female, Age, etc.)
Location
Network condition that is also location-based
Background apps
Non-full battery charge

Bottom Line: Match the Market and Tailor Your Test Coverage

For the most consistent and reliable testing, teams should match their testing with market usage patterns. This means tailoring test coverage that spans the geographies of their apps as consumers expect seamless app functionality across digital channels.

The good news is, with Perfecto, you don’t have to spend hours creating a test coverage strategy and methodology. We created a resource that will help you create a test coverage plan that will grow with you and your customers.

This is our proven test coverage index that considers all device and OS combinations across 13 different countries, paired with other characteristics like aging, screen parameters and other testing related guidelines for data-driven test coverage of the mobile market.

Download the Test Coverage Index

The 2023 State of Test Automation