Top Test Coverage Metrics for Flawless Apps
Test coverage metrics are important to consider. In this blog, we break down what test coverage metrics are, why test coverage analysis is important, and how to ensure test coverage.
Read along or jump ahead to whichever section interests you most:
- What Is Test Coverage?
- What Are Test Coverage Metrics?
- How Do You Measure Test Coverage?
- Why Is Test Coverage Analysis Important For Mobile and Web Apps?
- Software Test Coverage — Beyond Devices/OS
- Top Test Coverage Metrics (Examples)
- How to Ensure Test Coverage
What Is Test Coverage?
Test coverage measures how much your tests are covering things like test requirements, code, different user scenarios, and platforms such as mobile devices or browsers. It is a useful metric for measuring the effectiveness of your testing efforts. Test coverage is important for finding defects before they reach users.
What Are Test Coverage Metrics?
Test coverage metrics are used to measure and monitor your testing activity.
Test coverage metrics help you improve the testing process and maximize efficiency.
You can use test coverage metrics to get the answers to important questions, like:
- How much of the software was tested?
- How many bugs did the test team not find?
- How long will it take to test?
- Am I testing against the right environments/platforms?
But which test coverage metrics matter? Perfecto publishes a comprehensive mobile and web test coverage index biannually to help you understand.
This index includes:
- The latest annual web and mobile release calendar.
- Our proven methodology.
- The phones and browsers most used by your audience.
- Insights from countries across North America, Asia, and Europe.
Latest Test Coverage Index
Get your copy of the latest test coverage index today.
How Do You Measure Test Coverage?
There’s a simple formula often used to measure code test coverage:
Divide
The number of lines of code all test cases currently execute
By:
The total lines of code in the software you are testing
To get the percentage of test coverage.
In addition to code coverage metrics, it’s important to look into covering the right test scenarios. This provides the highest value from a feedback perspective as well as the right target platforms (web and mobile). These two considerations are two sides of the same coin.
Why Is Test Coverage Analysis Important For Mobile and Web Apps?
Test coverage analysis is very important for both mobile and web apps. Doing a test coverage analysis is the best way to prevent escaped defects.
That’s because…
Users Don’t Tolerate Bugs
Users today are simply less tolerant of buggy performance on the websites and apps they love.
Functional Testing Isn’t Enough
The agile testing manifesto advises to cover all types of testing within the software iteration cycle. To prevent various defects, you’ll need functional, performance, security, accessibility, and more types of testing.
App responsiveness and quality are critical to the digital experience and digital confidence, and add additional layers to testing strategies. This includes performing tests under real user conditions.
A Comprehensive Test Coverage Analysis Is the Only Way to Succeed
For app developers, this shift to a digital-first world requires a comprehensive test coverage analysis.
It asks for more rigor and accuracy in:
- How digital test labs are built and maintained.
- What platforms to include and how often to test on these platforms.
- What test scenarios to include per each software iteration and when in the pipeline to execute them.
All while dealing with rapid market changes.
Software Test Coverage — Beyond Devices/OS
Software test coverage goes beyond devices and operating systems. There’s more you need to consider as you develop your software test metrics.
For example, you’ll want to consider user conditions, such as:
- Switching networks.
- Apps running in the background.
- Phone call interruptions.
- Memory consumption.
- Sudden spikes in mobile traffic.
- Low battery.
Top Test Coverage Metrics (Examples)
The top test coverage metrics include the following examples:
- Test execution coverage percentage.
- Requirements coverage.
- Test cases by requirement.
- Defects per requirement.
- Defects per platform under test.
- Defects per functional area.
- Test case performance in finding defects.
How to Ensure Test Coverage
Here’s how to ensure test coverage and use the test coverage metrics.
1. Collect the Data
The best way to define and plan your digital test coverage is to have a mix of the most relevant metrics. This includes what devices and browsers people use to visit your app and website and who your competitors are. You should also keep an eye on market share data and device releases and browser updates.
The way we recommend structuring your test coverage is to combine the following sources into one Venn diagram.
In addition, your teams should gather test scenario history to guide your future software iteration test scoping.
We recommend ensuring that only high-value, robust, and cross-platform test cases are part of the pipeline and CI. This isn’t a one-time audit. Rather it’s a continuous monitoring of your test suites to ensure the regression suites and others are up to date.
2. Determine Device and Platform Criteria
Next, you’ll need to determine device and platform criteria.
At Perfecto, we recommend that teams include one or more platforms (smartphone, tablet, desktop browser) from each of the four groups below — no matter which test coverage goal they’re trying to meet.
Reference Devices
This is a key group because it includes devices such as Google’s Pixel. These devices are important because they will always be the first to get beta and GA versions of the Android OS. This allows dev and test teams enough time to test their apps on the upcoming platform. These
devices should be part of the test lab despite their market adoption or share.
Popular Devices
This group is a no-brainer to include in your test lab. It can consist of devices coming from both your customer data and the greater mobile market.
Legacy Devices
In this group, we find older iPads, Samsung devices, browsers, and mobile OS versions. These devices are popular in various markets and as such, require testing. However, they’re often slow to receive the latest OS updates. Also, running on older hardware with less CPU and memory can be challenging for modern applications that support newer features.
Emerging Devices
It’s imperative to treat your digital platform as an ongoing effort and therefore it’s crucial to keep an eye on new devices, operating systems, and other trends and be prepared to test them. This includes new devices or major beta versions of iOS or Chrome. Including these devices in the mix can save R&D time later on and also position your brand as ahead of the curve.
Once you are able to gather the above insights and make an informed decision on the test lab setup, you’re ready to think about the size of your lab.
The DevTest team should understand how many of the test lab devices and platforms they actually need to cover all manual, automation, performance, and other test cases. They also should understand whether these platforms are sufficient for the entire team, based on head count.
Sizing the lab correctly can positively influence your team efficiency by eliminating cases where developers or testers are waiting for a device or desktop to be released by another user. It can also help reduce your overall testing cycle time by executing more tests in parallel on more platforms.
3. Include User Conditions in Your Test Coverage
After you have the right mix of devices for your lab, you’ll need to think about the real life conditions your users experience each day.
- What networks do they use? Are they on Wi-Fi?
- How many apps do they run at once?
The digital platforms that consumers use are complex. You’ll want to recreate specific user environments as much as you can in your test lab.
You should also consider testing for these conditions on different screen orientations (portrait and landscape).
By taking a set of tests that run against a given number of devices and desktop browsers on a functional level and factoring in real user conditions, you enhance your test coverage. You add more depth to your testing and increase the likelihood of meeting your user expectations.
4. Define the Right Coverage Mix
Now it’s time to use the data you’ve collected, device and platform criteria, and user conditions to determine the right test coverage mix. This will help you build a lab with mobile devices and browsers.
Use the following testing considerations:
- Supported locations (countries).
- Supported mobile platforms (iOS, Android).
- Target test coverage level (Essential, Enhanced, Extended).
We also recommend building your lab with the following guidelines to achieve the optimal test coverage and the least risk:
- Test on the top 10 different device/tablet models on various OS versions, in what we call the “Essential” group.
- Follow up by expanding to a list of top 25 total devices, including the top 10. This second group, the “Enhanced” group, will represent an optimized list from the market’s longest tail of popular, legacy, and new device/OS combinations.
- From there, you can move to the third group for “Extended” test coverage that can be met by testing on 32 different device/OS combinations. In this group, we will either see devices that are older but still need to be tested against, or very new but not popular yet.
As part of each group mix, the following attributes should be considered:
- Device and OS popularity, including Custom OS versions (market share).
- Screen sizes, resolution, and other screen attributes such as pixel per inch (PPI).
- Device age (launch date).
- New and trending devices and platforms.
- Operating system version update rate (e.g., reference devices like Android Pixel get a higher score).
- Unique device properties important for testing purposes — chipset, CPU, memory.
- Audience demographics.
The second and third groups offer the highest test coverage and the least risk to digital teams.
You should also have an ongoing process of refreshing your test lab. This is usually done on a quarterly basis, to make sure nothing major has changed in your user base or in the market that would require a new device or OS.
Ensure Test Coverage With Perfecto
Test coverage doesn’t need to be daunting. With Perfecto, you’ll get the tools and tips you need to ensure test coverage — for every metric.
Perfecto’s cloud-testing platform gives you access to all of the devices, platforms, and browsers that you need to test your app on. You’ll be able to use this cloud-based lab to ensure proper test coverage for your devices, platforms, and user conditions.
Get started today with a free 14-day trial.
Ensure Test Coverage With Perfecto
