March 12, 2021

How to Do Single User Performance Testing With Perfecto

Automation

Continuous Testing

This blog breaks down client-side performance testing, also known as single use performance testing. Keep reading to learn how to do performance testing in the Perfecto cloud.

What Is Performance Testing?
Performance testing is a type of non-functional testing. It tests the quality of an app under varying capacities.

Performance testing is critical because it provides insight into the different aspects of an app, such as speed, stability, and scalability. Without it, users may encounter poor usability.

Where Performance Testing Typically Goes Wrong

Too often, performance problems arise late in the application lifecycle. The later you discover them, the greater the cost and effort to resolve them.

The planned line (solid) indicates the expected outcome when the application is developed with a carefully-factored process. This typical application deployment benefits the business immediately, as long as there are no performance problems.

The actual line (dotted) shows what more commonly happens — when development and deployment targets slip (striped diamond). Significant time and resources are involved in trying to fix performance issues in production. This is bad news for the business because the application fails to deliver the expected benefits.

Performance Testing Needs Real User Conditions

Traditional load testing does not put much emphasis on bottlenecks on the client.

Post-digital transformation, more than 60% of traffic now comes from smartphones. These devices have limited memory and CPU, with a wide range of specifications provided by a variety of vendors.

Also, these devices are exposed to environmental conditions and real user conditions that affect application performance such as:

Having to work in different Wi-Fi or cellular conditions.
Constantly switching between access points and cellular.
Calls and messages interrupting applications.
Applications running in the background.
Location changes as users are on the go.
And many more.

Overall, no matter how good a server and server infrastructure are, app performance depends on the client-side too. That’s where single user performance testing comes in.

Performance Testing With Perfecto

Perfecto, the industry-leading testing cloud for web and mobile apps, offers a mature performance testing solution. It emphasizes single user performance testing on the client side, supporting end-user conditions besides server-side load testing (third-party integrations).

👉 Try Performance Testing With Perfecto

All these capabilities are easy to add to your test and become visible in Perfecto’s test reporting and failure analysis, which is designed to help identify and drill into issues.

Perfecto provides:

Real user simulation
Browser front end metrics
UX timers
Transactions\checkpoints
Rich Artifacts — network HAR files, vitals per platform and transactions, KPI metrics, assertion reports, logs, videos and screenshots for efficient app debugging and resolution of UX issues
Third party performance testing tools integrations

How to Do Single User Performance Testing

Now, learn how to get started with single user performance testing.

Prerequisites

MITM enabled in the cloud and certificate should be installed in the device.
Capturing traffic for mobile application – Mobile applications will not allow MITIM proxy to intercept network traffic due to HTTPS requests. Please ask your developer to provide the unpinned version of binaries (IPA and\or APK ) to capture the network traffic.

Getting Started

Client-side performance testing, also known as single user performance testing, evaluates client-side KPIs like response time and device vitals.

Single user performance testing needs to take into account all of the factors that could impact the end-user experience across all tiers, including last-mile networks and client devices.

This includes network changes and signal degradation, changes in location, usage of onboard resources and sensors (which consume CPU, memory, etc.), background applications competing for device resources, etc.

All of these variables should be considered in the test scenarios and they should all be automated since the time to test is very limited. SLAs for acceptable user experience should be set (e.g., time to login, time to accomplish a task) and measured in all scenarios.

Related Reading: How to Decide Which Types of Test Cases to Automate

Single User Performance Testing Example

In the below example, we will evaluate Etihad Airways’ web application performance metrics.

We have selected two transactions:

Time taken to launch a web page.
Time taken to list the available flights on the given itinerary.

Perfecto’s real user simulation offers the following abilities to extend coverage to real user conditions when testing user journeys or flows. The test will have the following sections:

End-user conditions that simulate network, location, background applications, and other properties.

//Set End User conditions

//Throttle Network and Capture Traffic - In this example, HAR captured only for Android since Etihad IOS app expects trusted certificate

if(platformName.equalsIgnoreCase("Android")) {

Map<String, Object> pars = new HashMap<>();

pars.put("profile", "4g_lte_good");

pars.put("generateHarFile", "true");

driver.executeScript("mobile:vnetwork:start", pars);

}

else {

Map<String, Object> pars = new HashMap<>();

pars.put("profile", "4g_lte_good");

driver.executeScript("mobile:vnetwork:start", pars);

}

//Set location

Map<String, Object> params = new HashMap<>();

params.put("address", "Bangalore, India");

driver.executeScript("mobile:location:set", params);

//Run background application

String backGroundApps = "YouTube,Messages,Maps,Calculator,Calendar";

String[] bApps = backGroundApps.split(",");

for(String i: bApps) {

try {

Map<String, Object> params1 = new HashMap<>();

params1.put("name", i);

driver.executeScript("mobile:application:open", params1);

} catch (Exception e) {}

}

Points of interest that identify crucial application flows such as AppLaunch and Search Flight.

WindTunnelUtils.reportTimer(driver, AppLaunchTime, 10000, "Checkpoint load time of App launch.", "AppLaunchTime");

Timer reporting ability to track timing and performance of the application. For example, it could track how long it took for a page to load or a process to happen, and what that actually looks like for a user. In this test, we have chosen app launch time and flight search transactions.

Perfecto provides page load time as User Experience Time (UX) with the help of visual functions (OCR and image analysis). In other words, the timer will wait for specific text or an image to actually appear (be rendered) on the screen.

In our example, on launching the Etihad app, the web timer will wait for the text “One Way” to appear on the screen. Perfecto page load time will be comprised of:

Page Load Time = UX rendering time + API process time + network time.

NOTE: The reference text or image should be the last component on the page that was loaded.

//Launch Web application

driver.get("https://www.etihad.com/en-us/book");

ocrTextValidation(driver, "one Way");

// Wind Tunnel: Measure UX timer 1 - Able to retrieve UX Timer value

long AppLaunchTime = timerGet(driver, "ux");

System.out.println("'Measured UX time for App launch time is: " + AppLaunchTime);

private static void ocrTextValidation (RemoteWebDriver driver, String content) {

// verify that the correct page is displayed as result of signing in.

Map<String, Object> params1 = new HashMap<>();

// Check for the text that indicates that the sign in was successful

params1.put("content", content);

// allow up-to 30 seconds for the page to display

params1.put("timeout", "30");

// Wind Tunnel: Adding accurate timers to text checkpoint command

params1.put("measurement", "accurate");

params1.put("source", "camera");

params1.put("analysis", "automatic");

params1.put("threshold", "90");

String resultString = (String) driver.executeScript("mobile:checkpoint:text", params1);

The outcome of the script should be a detailed report that’s easy to understand and provide the below details:

Functional Validation — Feature is functionalwith detailed steps and recorded video.

Performance Validation

Page Load Time — Measurement of responsiveness of the application’s rendered page load time retrieved by visual analysis.

Network Traffic — Report including a detailed (HAR) network traffic log.

Device Vitals — Recorded device vitals to provide visibility into network, CPU, and memory consumption during the execution of the test script.

Network Traffic Capture and Analysis (HAR)

HAR (HTTP Archive Viewer) is a JSON file that contains a record of the network traffic between client and server. It contains all the end-to-end HTTP requests/responses that are sent and received between the client and server.

Using a HAR file viewer allows you to view HAR files. These files allow developers and testers to learn what actually happens when a transaction is executed and to help find performance bottlenecks and security issues in the original and third party code.

In this example, we have used HAR viewer, a free web-based tool showing a waterfall graph of all the calls with the ability to drill down to a specific request.

Real-Life Analysis Example

The above figure is network traffic for a banking wallet app, captured as part of single user performance testing to analyze the performance metrics for a login transaction below.

Number of API requests — Track intended/unintended API transactions.
Identify sequential/parallel request — For better performance, it is recommended to trigger APIs in parallel rather than sequentially, unless APIs are not dependent. From the above example, the third and fourth API are going sequential before the parallel requests. So, the system took 4.5 seconds initially. Sending those calls asynchronously reduced ~3 seconds of page load time.
API timings — Time taken to process each API call. Each API call involves network time (DNS lookup, connecting to server, sending bytes, receiving bytes), and server time (time taken by a server to process). This helps in identifying latency involved in client-side or server-side testing.
Page Load Time – Drill Down — Generally, server-side tools calculate page load time by summing up the processing time of all APIs on a related transaction, ignoring UX rendering time. Perfecto provides page load time as User Experience Time (UX) with the help of visual functions (OCR and image analysis).

Page Load Time = UX rendering time + (API process time + network time)

In the example above, the script reported 10.5 seconds as the page load time for the login transaction. Let’s drill down with the help of network traffic.

Req. Start Time from Beginning	API Timing	Server Time	Network Time	Parallel Request
0	1.9	1.49	0.41	FALSE
2.5	1.43	1.1	0.33	FALSE
4.97	1.72	1.28	0.44	TRUE
5	1.45	1.03	0.42	TRUE
5.05	1.53	1.1	0.43	TRUE
5.1	1.42	1	0.42	TRUE
6.91	1.53	1.11	0.42	FALSE

Table: Login Transaction API details (extracted from HAR files)

Network Time — Total time spent in the network to transfer and receive the packet.

Breaking Down Network Time

Network Time = Sum of DNS lookup + connecting to server + sending bytes + receiving bytes of each API (excluded parallel request, as it sent as a bunch).
Network Time = 0.41 + 0.33 + 0.44 + 0.42 = ~1.6s

Breaking Down API Process Time

API Process Time — Total time took to transfer packets until last byte reception from the server.
API Process Time = (Last API Start Time + Last API Completion Time) – Network Time
API Process Time = (6.91s + 1.53s) – 1.6s = 6.84s
So, the login transaction APIs took 8.44s to process, spending 1.6s in the network to transfer and receive the packet.

Breaking Down UX Rendering Time

UX Rendering Time — Total time took to render in UI.
UX Rendering Time = Perfecto page load time – (API process time + network time)
UX Rending Time = 10.5 – (6.84s + 1.6s) = 2.1s

Summing It All Up

So, the final breakdown of the reported page load time will be as follows.
Page Load Time = UX rendering time + (API process time + network time)
10.5s = 2.1s + (6.84s + 1.6s)

Get Started With Performance Testing in the Cloud

Single user performance testing identifies bottlenecks at the user experience level. It is a critical step to providing better experiences for all users.

A cloud platform, like Perfecto, can help you execute single user performance testing on all the devices you need — and automate at scale.

With Perfecto, you can also…

Apply real user simulation to tests.
Extend coverage for platforms and test scenarios.
Get detailed test reporting for fast fixes.
Execute at scale and boost automation ROI.

See for yourself how Perfecto can improve your performance testing. Get started with a free 14-day trial.

Try Perfecto

▶️ watch a demo First

The 2022 State of Test Automation