machine learning
March 5, 2020

Machine Learning in the New Age of Test Automation Tools

Scriptless Testing

AI and machine learning (ML) are some of the hottest topics in the tech industry and are continuing to make a huge impact on how companies test software. At TestCraft, we continue to discuss the rise of AI in testing, and how it’s become an important asset for QA professionals in any industry.

We not only discuss AI in testing within our own community, but our CEO and co-founder, Dror Todress, also had the recent opportunity to discuss this phenomenon in a guest webinar with Software Test Professionals (STP). In this webinar, Dror spoke thoroughly on how ML functions in different QA processes, with day-to-day testing examples and statistics. Here’s a short recap of what Dror had to say, as well as the questions he answered at the end of the session.


Machine Learning in Digital Transformation

Dror started off the webinar by reviewing the ins and outs of the digital transformation we are now facing. The major shift of software development from waterfall to agile has created a need for constant software updates and shorter release cycles. This need to shorten release cycles means that companies must prioritize continuous test maintenance by established an automation testing framework.

Yet since applications change continuously, the tests need to be maintained and updated by the QA team. In this case, AI and ML testing tools are critical for frequent releases and to assist the team during the development and release cycle. With a test automation framework that has ML capabilities, your testing team will be able to overcome those changes or easily adapt the tests to fit the change. By overcoming these changes easily, your QA team will feel empowered to catch defects before release, spend more time on different types of testing that require more innovation, and more.


The Importance of ML With Codeless Selenium

As mentioned earlier, machine learning and automation are critical to the agile method. Dror drilled down into our ML capabilities and how our algorithm works to automatically overcome changes in an application to keep tests from failing. He then went into live examples of how testing with AI works, showing the potential for how much time and resources testers can save on test maintenance.

He then shared with the STP community about how to determine whether or not the AI-based features of automated testing tools are actually working. Dror advised those in attendance to ask about the tools’ accuracy and misclassification rates, to make sure that the tool isn’t producing false positives or overcoming changes that actually require additional attention. He then showed how the TestCraft codeless Selenium can overcome 97.4% of the changes in the application (that were not caused by a change in the app logic or process), according to the company’s accuracy and misclassification rate data.


Confusion Matrix

How to calculate accuracy and misclassification rates for AI-based testing tools.

Questions From the Audience About AI in Testing

Dror closed the webinar with a Q&A session. 

Machine Learning in Test Automation

Q: Are there any other areas where you can use machine learning in test automation?

A: Yes, there are many areas where AI and machine learning can help with software testing. Some examples include results analysis or helping users decide which tests to run at which time. TestCraft’s specific focus today is using AI and ML to help with test maintenance; our self-healing technology overcomes changes automatically in the application and keeps tests from failing. By reducing the amount of time spent on test maintenance, AI and ML testing tools can also help improve your overall test coverage.

Q: How does machine learning in test automation work with Ajax and elements that appear and disappear on the screen?

A: This happens to be an area of focus for us. First of all, before we run a test on a page we wait for the page to fully load and then we wait for Ajax codes to complete before we run the test. In addition, we wait for elements to load completely in between steps, allowing us to support Ajax calls as the test runs. We also allow users to add explicit timeouts, but we advise against doing this if possible.

Q: You’ve shown many metrics and calculations that are taken into account as part of the ML algorithm. How does it affect runtime – does it increase the test execution time significantly?

A: It doesn’t slow down test runs.

How AI Works Within Testing Tools

Q: Currently my company is using Selenium for automating web applications. If we want to add machine learning to my test automation environment, what do I need to know? How can I use ML for my existing automation projects?

A: You will need to create some type of custom solution to add machine learning to your Selenium testing. This will most likely entail going through a long and costly process of hiring engineers that specialize in machine learning and instructing them to write the code. It is not a simple question that I can answer with general instructions.

Q: How many selectors do we need to use as the input to achieve a high level of accuracy? Or does the system locate the elements themselves?

A: You don’t need to set up anything with our platform. The system collects all attributes for each test element, although we do allow more advanced users to intervene. They can do this by identifying the weights for specific elements in an attribute. This helps us improve the results further. For example, in the case of a login button, you can safely assume the text will not change anytime soon. Therefore, you can direct the system to give a high weight to the inner text attribute of that element, such as ‘login’. Due to this weight classification, the system will continue to recognize the login button that says ‘log in,’ even if it changes its color ID or location on the screen. This improves the resilience of this step.

Q: How easy is it to add the ML algorithm to existing test cases, for example in UFT and regression tests. I want to automate the login page, how can I use the ML algorithm?

A: I can’t speak to how you can add machine learning to external platforms like UFT or existing tests that are purely code-based (such as using Selenium on its own). In these cases, you will either need to create a custom solution to deal with test maintenance issues or go into the test itself to resolve the issue manually. Our platform does not require you to do anything special to get the ML algorithm into action. In the example of a login process, you simply navigate to the application and perform the login process on the screen, which then saves as a data set. Once you click ‘submit’ or ‘login’ it will save and you will have a login process that is supported by machine learning. It only takes a few minutes to automate a simple process.

Q: Is there an algorithm to identify image patterns? In some cases, we can’t interact directly with elements and we need to use image recognition (OpenCV for instance).

A: We don’t use image recognition to identify elements, although this is something we plan to do in the near future. That being said, we would use visual identification as one attribute of the test elements and would let the machine set the threshold for which method to use in which test case. This would be instead of forcing the platform to only use visual identification in specific cases.

Q: There can be DOM Trees with thousand of elements and perform a lot of calculations to find an element. How do you work through this?

A: We do it efficiently by benchmarking 30K elements with less than 200ms of working on all the DOM. There’s no significant delay.

About TestCraft

Q: How do we manage sites that are more intricate with high complexities that drive changes using machine learning and codeless Selenium?

A: TestCraft is fit for both simple and complex web application testing, as we apply the ML algorithm automatically to every test step that is added to a test scenario. The testers will first need to define the test steps, using our simple visual methods of doing so. We will then translate those steps into Selenium-based code and apply our ML technology. This applies whether you have a React or Angular-based application, and whether or not you use complex test components such as iFrames. TestCraft has a wide range of customers in the banking and insurance industries (as well as others) that use machine learning and codeless Selenium with more sophisticated test automation processes

TestCraft Screenshots

How TestCraft handles complex test scenarios

Q: Please talk about the actual workflow involved in creating and running the test while using your platform, using a day-in-the-life example.

A: As soon as you log into the TestCraft platform you have an online canvas that you can set for a specific test and open up the basic file (canvas) to create a flowchart. That flowchart consists of different elements and actions that represent elements on the application you are testing. By connecting them in a certain sequence, you create your flow. Now, there are different ways to create a flow: 1. Drag and drop elements on the canvas and 2. Launch a session on our platform and the tested app and go through the process you want to create (point and click), so you can actually click and assign an action on it (validation, double click, etc). Once you do that, the flowchart’s code is being written for you in the background.

Q: Can we use TestCraft’s ML capabilities to analyze a defect in the platform and make our automation more efficient?

A: Not with our platform at the moment. We do give you tools for doing that so you can analyze different executions for test suites over time. The tools allow you to compare and identify patterns leading you to improve the quality of testing.

Q: Are there any features in Selenium, i.e. any testing that can be done with Selenium but is limited for TestCraft?

A: Not that we’re aware of. Using our “Selenium Bricks,” TestCraft can take any Selenium library, wrap it up, and include as a test step, in case you come across a capability that we haven’t yet developed.

Q: Does your ML algorithm cater to automating performance and security testing scenarios?

A: Our focus is on end-to-end functional testing, not load testing at the moment.

Q: Is there a benefit of using your tool for a legacy application where element selectors do not change?

A: Absolutely. There are other benefits to using codeless test automation, such as simpler test creation and execution.

Editor’s Note: The post was originally published in November 2018 and has been updated for accuracy and comprehensiveness.


Explore TestCraft

 Learn about TestCraft, the codeless UI test automation tool for web apps. 

Explore TestCraft