Testing Pyramid Strategy for Automation
To successfully release an app out into the world, it needs thorough testing. And to successfully test an app, it needs a solid test automation strategy.
You probably already know that you don’t automate all test cases. You concentrate your testing efforts on some types of tests more than others. And the platforms you use differ depending on the type of testing.
All of this is mapped out in the testing pyramid. Use this blog as a guide to match the proper platforms with the right types of testing within your pipeline.
What Is the Testing Pyramid?
The testing pyramid is a concept that strategically groups software tests into different categories.
From top to bottom, the main layers of a testing pyramid include:
- UI/exploratory tests
- Integration tests
- Unit tests
The lower portion of the pyramid is more automated, and thus faster, while the top of the pyramid is slower and more manual. The higher you go on the pyramid, the fewer the tests.
In other words, teams should have many unit tests, which are small and isolated, and not so many end-to-end exploratory tests, which are time-consuming and manual. Frontloading tests at the unit level makes fixing bugs quicker and easier, as opposed to defects that are discovered much later in the process.
Now let’s walk through each layer of the testing pyramid, working from the bottom up.
The Testing Pyramid Base: Unit Testing
The base of the testing pyramid is unit testing. The scope of unit tests is small — and they ensure that isolated units of code work as they should. Unit tests should test one variable and not rely on any external dependencies.
Mobile App Testing Strategy Combining Virtual and Real Devices
In this layer of the testing pyramid, tests are executed per and post commit. Tests are dev triggered. For these tests, it is recommended to use virtual devices — simulators and emulators. Executing these tests on virtual devices is sufficient and provides the necessary feedback and debugging/log information. Plus, virtual devices offer the benefits of being faster and more cost efficient for testing.
The Middle Stage: Integration Testing
In the middle of the testing pyramid, you test across integrations of more than one system. Unlike unit tests, these tests are CI-triggered. In this stage, you have more features to cover and more testing types to conduct. This is the stage where it is critical to get the right balance of real and virtual devices.
Testing Apps on a Simulator vs. Emulator vs. Real Device
Some types of testing, such as acceptance testing, can be done on virtual devices. However, other types of tests, such as performance testing, security testing, and certain accessibility features, cannot be done on virtual devices, so it is best to use a mix of real and virtual devices at this stage.
You will also want to add additional real device/OS configurations that match the client usage analytics and market trends. You can get device/OS usage data in our Test Coverage Index.
Using both real and virtual devices in your testing strategy is a best practice. Real and virtual devices complement each other and support your goal of achieving high-quality builds for nightly regressions.
The Top of the Pyramid: UI & Exploratory Tests
Once you get to production monitoring at the top of the testing pyramid, testing must be done on real devices. Why? End-to-end testing, as well as exploratory testing, requires real devices to more closely mimic the app on a user’s device — because real users report defects on real devices, not on virtual ones.
It is also critical to apply real user simulation to your tests. Testing with conditions such as conflicting apps, call/text interruptions, network throttling, network latency, and more enable you to see how your app will hold up out in the world under varying circumstances.
And, keeping in mind the shape of the pyramid, the tests in this section should be far fewer than those at the base. That’s because these tests take longer, are more complicated, and have more dependencies.
Apply the Testing Pyramid Strategy With Perfecto
The testing pyramid strategy doesn’t need to be daunting. Make it easy with Perfecto.
Perfecto’s cloud gives you access to all of the devices, platforms, and browsers that you need to test your app on. You can use the cloud to ensure proper test coverage across platforms. You’ll even be able to apply environmental conditions by using real user simulation.
Plus, with Perfecto, you’ll be able to execute on your testing strategy at scale — and deliver true ROI to your organization.
See for yourself how Perfecto can improve coverage across platforms. Get started with a free 14-day trial.