The Current State & Future Trends of AI in Software Testing
Artificial intelligence (AI) and machine learning (ML) are transforming multiple sectors of the economy, and impacting several aspects of our daily lives.
Workplaces such as those in finance, healthcare, retail, education, and technology are leveraging AI to automate tasks, reduce costs, and make data-driven decisions. In our homes, AI is powering television and movie recommendations, personal digital assistants, security cameras, and home automation.
AI in Software Testing
As AI continues to permeate our world, it is becoming more and more critical to validate that these types of systems are functional, safe, secure, performant, available, and resilient. In other words, AI needs testing. Unfortunately, we haven’t seen many advances in the area of testing AI-based systems.
On the bright side, researchers and practitioners are recognizing the potential for AI and ML to bridge the gap between human and machine-driven testing capabilities. As a result, several organizations are developing AI-powered automated testing tools.
Since 2014, there has been a spike in the number of vendors offering AI-driven testing services. The majority of these tool vendors are start-up companies targeting system-level testing of mobile applications, and the subject is generating some much-needed buzz in the industry.
Hence, it seems that just like AI needs testing, testing needs AI.
Startups and vendors in the AI for testing space
Although some aspects of AI for Software Testing (AIST) are still not receiving enough attention, in the last decade we have seen it emerge as a discipline centered at the intersection of three areas:
- AI-driven testing — developing AI tools to test software.
- Testing AI systems — devising methods to test AI systems.
- Self-testing systems — designing software that is capable of self-testing and self-healing.
The intersection of AI and software testing
AI in Software Testing: Current State
The current state of the practice uses autonomous and intelligent agents, referred to as “test bots,” to automate activities such as application discovery, modeling, test generation, and failure detection. A combination of machine learning techniques are used to implement test bots. These include but are not limited to decision tree learning, neural networks, and reinforcement learning.
Machine learning allows the test bots to be robust and act under conditions of uncertainty unlike traditional test automation tools and frameworks. Examples of AI-driven testing approaches that have formed over the last decade include:
- Differential testing — comparing application versions overbuilds, classifying the differences, and learning from feedback on the classification.
- Visual testing — leveraging image-based learning and screen comparisons to test the look and feel of an application.
- Declarative testing — specifying the intent of a test in a natural or domain-specific language, and having the system figure out how to carry out the test.
- Self-healing automation — auto-correcting element selection in tests when the UI changes.
AI in Software Testing & the Open Source Community
Element location using AI has been incorporated into the open-source mobile testing framework Appium. The project is a collaboration between Jason Arbon, CEO of Test.ai, and Appium’s creator Jonathan Lipps. It uses AI to identify web elements in mobile applications.
Leveraging AI for web element location makes the tests quicker to create and less brittle. Furthermore, the testing community can improve the AI by adding new training data, alternative training methods, more rigorous relevance testing, or new labels.
The training data is open and available on Kaggle, thereby allowing individuals or organizations to fork the data, clean it up, add it to their proprietary test frameworks, or compete to improve the classifiers.
Tariq King and his team over at Ultimate Software developed and released an open-source framework called AGENT — AI Generation and Exploration iN Test.
AGENT is an ML-based, multi-agent prototype for web application testing. The prototype consists of a collection of intelligent agents that work in conjunction with abstract test generation and web UI classification subsystems. The agents incorporate long short-term memory units trained by human testers.
For any given subsequence of steps through the application, the agents predict sequences of the next possible set of actions and assertions that a human tester might make and automatically execute them on the system under test.
Future Trends for AI in Software Testing
One of the trends that started this past decade, which is expected to continue, is the use of AI to enhance existing tools and frameworks that target specific testing problems.
Examples at this time include functional testing of web and mobile applications, visual testing of user interfaces, and UI element location and auto-correcting element selectors. Beyond this we should expect to see AI starting to replace entire technology stacks for automated testing. At all testing levels, AI will take over automation tasks that require decisions that a human could make in less than a second. Initially, higher order testing tasks may still require human input or intervention. These are the tasks which require a bit more thought such as test generation, usability testing, security testing, and edge cases.
However, over time as technology progresses, and as the machines are trained on the actions of these higher order tasks, AI is likely to take over those activities as well and tackle problems that require deeper context. The image below depicts this progression of the AI testing singularity as we move into the next decade.
The AI testing singularity: a projection of the next decade of software testing
How Can We Prepare?
So what can testers do to prepare for a future in which AI tests software? The initial step is to determine whether you are interested in designing and developing AI-driven systems, or being an end user of these tools.
An engineer who is building these types of systems needs to have a good understanding of AI and machine learning. Both MIT and Stanford University have freely available introductory courses on machine learning.
Tariq King and Jason Arbon have open-sourced the materials for an introductory course on AI and ML skills for the testing world, including the slides and tools used during the exercises. It is unlikely that end users of AI-driven testing tools will need to have deep knowledge of machine learning. Vendors are likely to provide easy-to-use interfaces and APIs for using AI-driven testing features and customizing pre-trained AI models.
This trend is already being followed with the advent of AItools and platforms like Google’s AutoML Vision, H2O.ai, and DataRobot.
Bottom Line
AI is already altering the landscape of testing. And while we don’t know exactly what the future of software testing holds, we can prepare for it by stabilizing and scaling test automation to mature our processes.
Perfecto can help you take your test automation to the next phase, no matter where you are at.
Whether you’re just beginning on automation or you’ve reached continuous testing, Perfecto has the end-to-end testing support to help you accelerate. Try it for free today.
