Blog

May 27, 2020

Automated Visual Testing for Mobile and Web Applications

Dmitriy Yanovskiy
Dmitriy Yanovskiy

In the digital era where people have different devices to access websites and applications, most companies have testing teams to ensure their services meet the desired requirements.
Here, learn about automated visual testing and how it can help you provide better digital experiences.

Common UI Defects

Negative experiences with a product may be enough for the user to never come back and use the service again. On a daily basis, many organizations face UI issues on their web or mobile applications. 

The most common types of defects are:

  • Unknown marks instead of text.
  • Alignment.
  • Out of bound and images.
The referenced media source is missing and needs to be re-embedded.
The referenced media source is missing and needs to be re-embedded.

Most escaped defects to production are UI problems. UI issues escape because most organizations use standard Appium/Selenium drivers without visual commands. That’s because the standard Appium/Selenium driver doesn’t support it.

Due to a lack of practices around automated visual testing, teams tend to rely on manual testing. The more mature the DevOps organization is, the less manual testing will happen. This causes more escaped defects due to time restrictions and a lack of automation.  

What Is Automated Visual Testing?

Automated visual testing is a type of testing that’s done to ensure UI appears as it should in a web or mobile app. And, it can be automated to save teams time.

Visual analysis functions like FindText, FindImage, etc. use the executeScript() driver method. These functions allow the user to combine Appium/Selenium generic commands with text or image recognition commands.

By using the Optical Character Recognition (OCR) algorithm, users can find and verify any text or image, no matter what screen resolution they are working on. OCR is for text. Image/pattern matching are for images. 

What is Optical Character Recognition?

Optical Character Recognition is a way to translate visual representation of characters, words, and sentences (i.e. pixels) into text. OCR uses AI to analyze the shape of the characters, context to other characters, words and general sentence for correctly translating pixels into text with high accuracy. 

How Does Optical Character Recognition Work?

First, the program analyzes the structure of the device screen, either mobile or desktop screens. It divides the screen into elements such as blocks of texts, tables, images, etc. The lines are divided into words and then into characters.

Once the characters have been singled out, the program compares them with a set of pattern images. It uses hypotheses about what each character. Based on these hypotheses, the program analyzes different variants by breaking lines into words and words into characters.

After processing a huge number of such hypotheses, the program finally makes the decision, presenting you the recognized text.

Automated Visual Testing Can Help

Is it enough to perform the validation by identifying the UI objects based on XPath, UI automation, or similar methods? Do we need a human eye to complete automated tests? Can we combine these two actions and perform E2E automated tests?  

The answer is yes.

If you’re unsure how validation fits into the broader QA process, it’s helpful to understand the difference between verification and validation testing. By using text and image recognition, we can achieve more than just finding and clicking on UI objects (based on XPath, CSS, etc.). We can add human eyes on top of each screen.

Text and image recognition must become an extension to your Appium/Selenium scripts if you want to reduce escaped defects.

Automated Visual Testing With Perfecto

Let’s take as an example a responsive banking app. Before performing any user actions on the screen, like clicking on one of the icons/menus using a standard Appium method/command: 

driver.findElementByXPath("//*[@id=\"navCheckingMobile\"]//*[@class=\"title\"]").click();

The referenced media source is missing and needs to be re-embedded.

I want to verify several items on that page in this exact order: 

  • Checking
  • Auto Loans
  • Savings    
  • Small Business
  • Credit Cards
  • Investing

I can do this by doing a visual exam of my screen using ad advanced OCR algorithm. Here is a text validation command for finding a multiple text objects:

//Text Validation command, finding 6 different items on any screen resolution 

Map<String, Object> bofaMenu = new HashMap<>();
bofaMenu.put("content", "Checking AutoLoans Savings SmallBusiness CreditCards Investing");
bofaMenu.put("target", "all");
driver.executeScript("mobile:checkpoint:text", bofaMenu);

Learn more about visual analysis with Perfecto here.

Try Automated Visual Testing With Perfecto

Experience fewer escaped defects with Perfecto.

With extended test coverage in the cloud, you’ll have access to all the mobile devices and browser VM versions you need to test against. And with advanced automation capabilities, like automated visual testing, you can automate even more than ever.

See for yourself how Perfecto can help you automate testing.

Get started with a free 14-day trial.

Try Perfecto

Related Content