Leveraging Artificial Intelligence & Machine Learning in Test Automation

Introduction: Integrating Machine Learning into Software Testing

Machine learning (ML) is an exciting, new field that is changing our relationship with technology as we know it. From improving home security to creating smart digital personal assistants, machine learning helps technology develop artificial intelligence (AI) capabilities that can benefit us in the short and long term.

In the world of software testing, machine learning is also making a strong impact. With the digital transformation of all major industries, manual testers are constantly looking to speed up their work to keep up with their companies’ shorter release cycles.

While many testing and QA teams have incorporated test automation into their environments to meet these needs, ML is taking the efficiency and speed of manual testers to the next level. This not only helps manual testers better excel at their jobs, but also helps other testers and engineers on their team produce higher quality work in a shorter amount of time.

This eBook will delve into how test automation tools are incorporating machine learning into their platforms, and how software testers can leverage it to adapt to this digital transformation. When done successfully, QA teams can ensure that their work is both high-quality and high-quantity to support their companies’ agile and DevOps processes.

Background: The Challenges of Selenium and How Codeless Test Automation Can Help

Before diving into machine learning, it’s important to understand what prompted this shift to test automation in the first place. It all starts with the age of digital transformation.

The Effects of Digital Transformation on QA

Companies in all major industries, from banking to healthcare and e-commerce, have undergone some form of digital transformation to provide a better user experience for their customers. Consider the banking industry. Actions that used to require going to the physical branch, such as bank transfers or deposits, are now happening online with a few clicks of a button.

This digital transformation caused a few things to occur. Companies were tasked with creating and releasing frequent updates to their software. Additionally, they were also required to create a flawless user experience, for fear that their customers would leave them for their competitors. To fully embrace this digital transformation, many companies switched from waterfall to an agile methodology of work to keep up with their constant updates and versions.

Much of digital transformation focused on developers and DevOps processes, ensuring that they had the most up-to-date tools and methods to keep up with an agile process. On the other hand, manual testers and other QA professionals often had to work with legacy tools and couldn’t move quickly enough to meet company needs. This transformed testing into a bottleneck area, where QA teams were struggling to maintain their high quality of work while increasing their testing speed.

Selenium and Codeless Test Automation

A major way that QA professionals tried to combat this phenomenon was by adopting Selenium as the framework for their test automation needs. As an open-source and well-known platform used by testers worldwide, this seemed like a great way for testers to come of age in an agile era. In fact, Selenium has since become the most common open-source code with a steadily growing user base worldwide.

But many QA teams find that while Selenium is open-source, it has other hidden “costs” when it comes to actually using the tool. Selenium is difficult to set up and maintain without coding knowledge. Plus, it creates a huge learning curve for testers to gain the skills required to work inside Selenium effectively. For many companies, the time and resources it takes to make the investment in Selenium worthwhile make it a less than viable option.

Codeless Selenium test automation addresses these challenges in a variety of ways. It allows for QA professionals to create resilient and maintainable tests while leveraging the capabilities of Selenium, all without needing to write code (although 100% codeless tools offer an option to incorporate code into their tests as needed). This paves the way for QA professionals to produce quality test flows at scale in order to keep up with shorter release cycles.

In addition, test automation tools are also incorporating machine learning into their platforms to make them even more agile and efficient.

What is Artificial Intelligence and Machine Learning Anyway?

For many, the phrases “machine learning” or “artificial intelligence” conjure up the notion of exciting developments in technology: self-driving cars, highly attuned facial recognition software or better online translations of text to different languages. Yet for others, these phrases evoke a feeling of fear, especially in relation to job security. With machines doing all of the tasks that make up our day and learning how to do them better over time, will our jobs even need us in the near future?

Source: Waitbuywhy.com

While this can be the case for some areas of work, AI and ML often do not mean that you’re losing your job to a robot. According to Wikipedia, machine learning is, “a field of computer science that uses statistical techniques to give computer systems the ability to ‘learn’ with data, without being explicitly programmed”. There are many possibilities that will not only allow you to keep your job but also allow you to excel in your work.

Thus far, machine learning has made the most impact by taking charge of more repetitive tasks that need to happen on a frequent basis. This frees up your time to do the more complex projects that you were actually hired to do instead of occupying your days with mind-numbing busywork.

Practical Applications of Machine Learning for Web Testing

With Selenium for functional web test automation, QA teams often struggle in the following areas:

Test flakiness and website object management
Test code maintenance as the product evolves
Desktop platform and code coverage analysis
Generic reporting and release decision making

While each task on its own is time-consuming, they can become a huge bottleneck where these tasks are recurring for each sprint. With machine learning, QA professionals no longer need to concern themselves with these more mundane and time-consuming tasks.

Test Flakiness & Object Management

When a test case is bad by design, or it uses a wrong object identifier (flaky XPath, etc.), it carries a huge amount of quality debt per job within the continuous integration (CI) or regression cycles.

With machine learning, a test automation tool can scan through these test cases and both identify and resolve the issues that occur automatically. This makes tests more resilient, saving precious time and resources.

Test Code Maintenance

Ongoing test code development and proper object allocation is one thing, but what happens when the company adds a new functionality to the web product? Such changes automatically translate into adding new test code to existing scenarios, as well as adding new objects or elements to the site.

An ML engine that continuously scans your website can highlight these changes quickly, identify the gaps and develop black-box exploratory tests with the relevant objects identifiers. This allows for simpler test maintenance that doesn’t require going manually through all of your existing test scenarios to make these necessary changes.

Platform Coverage for Mobile and Desktop Web Browsers

While most of your time will be focused on optimizing test code and their respective objects with machine learning, knowing which platforms to test against as the market evolves is a large pain point on its own. With machine learning solutions, teams can figure out the most error-prone platforms (e.g. Windows 8.1/Firefox), and guide the testing teams to focus more on these platforms instead of wasting time on more robust platforms.

This capability should be well aligned and monitored continuously by QA teams, as the stability of platforms changes from one mobile OS/Desktop browser to the other.

Generic Reporting and Release Decision Making

At the end of each sprint in a DevOps process, the management team needs to decide whether or not they can release the product based on quality issues.

Having the entire pipeline visible from a quality standpoint at any given time and tied to the above pillars of robust and reliable testing on the “right” platforms is the last step in a mature machine learning framework for a DevOps practitioner.

By aggregating all the test data for all platforms that are supported for your website, ML-based test reports can give a QA team more confidence and the ability to make more data-driven decisions.

Machine Learning in Test Automation

We just determined how automated testing tools that have an artificial intelligence component can eliminate many repetitive aspects of software testing. By incorporating machine learning into their platforms, test automation tools make the QA teams that use them more productive and able to quickly adapt to changes.

But how does it actually work? What does the machine learning element of an automated testing tool actually look for to make these changes and learn about them over time?

If you’re conducting Selenium test automation for web applications, you know that each web page has a plethora of elements, such as a search bar, company logo, and buttons to click to other pages on the site. Each element is made up of multiple attributes, such as ID, CSS path, position, size, and value.

Machine learning provides an innovative way to answer the following question: “Which element am I looking for when running a test?” Using statistical techniques, machine learning enables testers to find the correct elements of a test both quickly and accurately.

‘Smart Binding Algorithm’: Identifying Elements in Test Scenarios

TestCraft’s AI-based algorithm, known as our ‘Smart Binding Algorithm’, combines codeless test automation with machine learning. It incorporates two different algorithms to identify an element – technical HTML attributes and weight distribution algorithms.

By attributing more weight to the less fragile attributes of an element, the algorithm allows for automatic updates in your test flow. In principle, the user should not have to touch these attributes, but in some cases, if the users wish to they can do so to enhance it. This makes tests more resilient, reducing the time and resources needed to do proper test maintenance.

Consider the following example. One of the most important elements on any website is the login button. It allows for personalized experiences that enable customers to make various transactions on the site. Many basic login buttons are made up of the following attributes: a username box, a password box, and a button that you click on to actually log in to your account.

While these are essential functions that allow a site to work properly, many websites often make subtle changes to their login element’s attributes so that their users can access and use it more easily. For this example, imagine that the website developer added a “remember me” checkbox for the next site iteration. This means that in this new version, the login button has changed shape and position.

Without machine learning, a tester would have to go into the test flow and manually identify and change the test scenario to account for this new attribute. When applied to hundreds of tests across dozens of website elements, maintaining these test flows and ensuring that they don’t break when attributes change becomes a very time-consuming task.

Alternatively, TestCraft’s Smart Binding Algorithm attributes less weight to these less critical attributes like a “remember me” checkbox. This allows TestCraft to overcome this change automatically and prevents the test of this login element from breaking without any human involvement.

In short, AI-based maintenance ultimately helps build more resilient and stable tests. With an increasing amount of test data, machine learning can also adapt to these tests and learn about which attributes of a test are most likely to change over time. This helps QA teams not only become more agile but also adopt a “shift left mindset,” allowing them to test even earlier in the software development process.

Does Machine Learning Actually Work?

These are two critical points to consider when starting to think about incorporating machine learning into your environment: its accuracy rates and misclassification rates. While accuracy rates refer to the amount of true positive and true negative results you get when testing, misclassification rates measure these tests’ false positives and false negatives.

Simply put, test automation with AI-based maintenance is only helpful if it identifies the right elements (high accuracy rate) and doesn’t produce false positives or negatives (low misclassification rate). You can use the two equations below to measure the accuracy and misclassification rates of any AI-based testing tool.

According to data from TestCraft’s customers that we incorporated into these accuracy and misclassification rate formulas, TestCraft’s machine learning algorithm accounts for changes in their web applications with 97.4% accuracy. In other words, in 97.4% of test scenarios, TestCraft is able to either overcome a change and identify the correct element or correctly identify a bug and cause the test to fail accordingly. This emphasizes the large impact that machine learning can have on test automation, and how it makes adopting test automation practices even more lucrative.

As for the remaining percentage of tests that don’t respond well to machine learning, it’s important to make the following disclaimer: Codeless test automation doesn’t replace all types of tests. There is no such thing as 100% automation, and there are certain types of tests that still should be conducted manually, such as exploratory testing.

Yet for regression and continuous testing, test automation will help QA keep up with shorter development cycles and ever-increasing product versions. With the addition of artificial intelligence and machine learning, test automation will focus less on maintenance and more on creating more complex and lucrative test flows for your company.

How to Choose a Machine Learning-Based Tool

This eBook has discussed the rise of machine learning, and how it improves test automation in an era of digital transformation. When it comes time to choose a machine learning-based tool for your company, it is critical to think about two main points.

First, you should determine which issue you would like the tool to solve, and whether or not the machine learning components of that tool can address this concern. TestCraft, in particular, uses AI to improve test maintenance and reduce the flakiness of tests. This is very beneficial for testing and QA teams who are looking to keep up with frequent releases while still producing high-quality work, It also helps teams that are struggling with maintaining their existing test automation and find themselves spending too much time and effort on these efforts, rather than on the actual tests.

Once you have determined this, you should always ask for the tool’s accuracy and misclassification rates. Any test automation vendor should be able to calculate these easily and supply you with the relevant information. With this information on hand, you can then feel assured that you are selecting the right machine learning-based tool for your company.

The Future Is Now: Mobile & Web Application Testing With AI