View all web browser and mobile devices available in our cloud-based test lab.
Performance testing does not resemble what it used to be, say, five years ago. When you examine the tools provided and their capabilities, they tell a story. Careful examination and tuning of the scalability, potential break points, and efficiency of backend services was the main objective. What changed? Here are a few pointers to get started:
As performance testing becomes more complex, it helps if we break it up into two practices: Single user performance testing, and multi-user load testing.
Single user performance testing needs to take into account all of the factors that could impact the end-user experience across all tiers, including last mile networks and client devices: network changes and signal degradation, changes in location, usage of onboard resources and sensors (that consume CPU, memory, etc.), background applications competing for device resources, etc.
Figure 1: Elements outside your app can impact the user experience
All of these variables should be considered in the test scenarios and they should all be automated since the time to test is very limited. SLAs for acceptable user experience should be set (e.g., time to login, time to accomplish a task) and measured in all scenarios. Ideally, many of the user conditions (background apps, sensor usage, location, network conditions, etc.) can be grouped into a “persona” that represents a certain type of user as defined by the marketing team or the line of business. The persona can then be used as a parameter to the script, and overlay user conditions on top of the functional test flows.
Figure 2: A persona (“Georgia” in this case) can be incorporated into test scripts as a parameter
The outcome of the script should be a detailed report that’s easy to understand and compare across browsers and devices, including the steps, measured KPIs (response times, resource usage), screenshots, and a video recording of the execution. Specifically, the measurement of responsiveness of the application is important and can be done using visual analysis:
Figure 3: Measuring the time to launch the Starbucks application
Device vitals are valuable to record since they provide visibility into network, CPU, and memory consumption during the execution of the test script. Furthermore, the report should include a detailed network traffic log: PCAP (packet capture) file to review packet retransmissions, and a HAR file to optimize network utilization by the app.
Figure 4: A waterfall chart based on recorded HAR file
While easy to integrate load into your script (ex.: Taurus), Load testing requires a setup that can sustain and support a high load on the environment. A typical dev/test environment is not powerful enough to simulate production and it is difficult to use it for gaining the insight sought. It is recommended that in this scenario, high load be applied on the service APIs, backend server architecture, etc., using virtual users that make direct calls to the backend web or application servers. At the same time, parallel execution of real devices on real networks should be used for measuring the true end-user experience while the backend is being stressed.
Figure 5: Load and user experience testing- architecture perspective
To fit performance testing into the Agile delivery cycle, we increasingly see organizations conducting user experience testing on a weekly or even nightly basis. Certainly the number of executions grow, and there’s more data to analyze. Performance expertise is being introduced to the application/feature teams for developing more efficient code and streamlining and accelerating the testing. To validate many variations in the application responsiveness, tests need to run in parallel on a robust execution grid. The tool we typically see being used for user experience testing is the “Wind Tunnel”: a cloud-based, persona-driven single user performance testing envelope. Load testing, on the other hand, is still typically performed less frequently and sometimes by a different team as it requires a performance-like staging environment to run on and additional expertise in building load test scenarios combining virtual users with real devices and a mix of scripts that represent production activities by different personas.
Figure 6: Single user performance shifts into the cycle
Interested in giving Perfecto a try? Enjoy the power of the industry's best testing platform for free.