Digital Testing Guide Part 1: The Challenges
In the past, organizations were able to easily define their target devices and platforms on which to test their products. But today the reality of digital testing is much more complicated.
The mobile and web markets are constantly changing, making it hard to predict and define a digital testing strategy. As an example, in 2015 alone (see image below) there were more than 30 significant new device launches and around 20 mobile operating system releases. In the desktop browser space, we saw about the same number of OS releases in 2015. Most of these releases are auto updated on the users' machines, adding more complexity to your digital testing strategy.
The complexity around test coverage is not only caused by devices, browsers and OSes; there are other factors, including:
- Types of tests (functional, regression, usability, performance, etc.)
- Hardware considerations (CPU, memory)
- Screen properties (size, resolution, pixel per inch (PPI) and screen orientation)
- User conditions (networks, background apps, locations, etc.)
- IoT (Internet of things) device communication
It's also important to understand that given the limits of time and budget, organizations cannot cover every browser/OS combination and therefore need to clearly define what digital testing is most relevant (to their customers, geography and industry).
Coverage as an organizational competency
Organizations often consider "test coverage" as solely a QA team responsibility. However, it's important to understand that getting the right lab coverage should be a priority for marketing, business development, product, QA and developers because all of these teams lose when app glitches make it to market. On the other hand, when test coverage is thorough and app quality is high marketing and business teams benefit from customer satisfaction as well as growth and market expansion. Meanwhile, dev and testers will have a much more focused set of platforms to develop and test against, resulting in a reduced number of production defects and fast feedback to developers for fixes.
Organizations need to invest in collecting data that's actionable and enables DevTest teams to make the right decisions. And because the digital space always involves moving parts, data collection and test lab coverage requires constant refresh.
The best way to define and plan your digital test coverage is to use the right mix of data from internal sources within your organization (customer analytics) and externally from the market (market share data, competitor insights). We recommend that you combine these data sources into one list of target platforms that your dev and test teams should focus on. This lab setup will serve as the base for future testing, so getting it right from day one will make it easy to modify.
Combining multiple data sources helps achieve several goals, including:
- Validate internal data
- Learn from the market and fine-tune where needed
- Gain visibility into what devices and OS versions are coming in the near future
Building a digital test lab involves many data sources and considerations. It's not just about desktop browsers and mobile platforms, but also about the environments in which they operate. And because digital testing is so complex, insights about customer and market data need to be visible to everyone involved in the delivery chain -- from marketing to IT ops to DevTest. Everyone has a hand in releasing a great digital product to market.