View all web browser and mobile devices available in our cloud-based test lab.
Testing in the era of the waterfall methodology was relatively simple. There was a single defined testing cycle which usually involved one group per test result. There were fewer digital platforms to test. And the window for testing was much wider.
The era of DevOps, also known as “continuous everything,” requires a totally different testing pace. Testing cycles happen multiple times a day, with multiple teams and stakeholders pushing code simultaneously.
There are 20-50 times more platforms included in test coverage metrics on average today. And the release windows vary from 3-4 hours to daily release up to 1-3 weeks.
As you can see, that’s a lot of testing to cover in a much shorter timeframe.
Table of ContentsNew Opportunities & RisksWhat Can Be Done? The Future of Analytics: Fail Fast, Fix FastHow Test Automation Will Become SmartBottom LineRelated ContentTable of Contents1 - New Opportunities & Risks2 - What Can Be Done?3 - 4 - The Future of Analytics: Fail Fast, Fix Fast5 - How Test Automation Will Become Smart6 - Bottom Line7 - Related Content
Continuous testing and test automation at scale are the answer for this fast-paced DevOps ecosystem. But these new methodologies also bring along massive amount of test result data. This sea of data, just like any other sea, creates opportunities and risks.
The risks include:
These risks prevent teams from expanding their test automation beyond the point they’re at. If you cannot analyze your test results and find defects in a reasonable time frame, what’s the point of adding more automation?
The opportunity here is to utilize technology for an efficient way of analyzing test results.
Teams also need to ask the following questions:
The increasing level of visibility into testing allows leaders to fail fast but to also fix fast. This is where testing starts generating value based on smart analytics. With visibility, managers can measure test automation success and identify critical regression testing problems fast enough to impact quality within the cycle.
And this is all happening as machines become more helpful to human-based software testing. AI in software testing is advancing. The timing is perfect — the only thing that experts can agree on is that testing will probably get harder.
To that end, AI and machine learning (ML) will offer four big promises around the ability to:
AI and ML will be able to analyze large sets of data fast and identify patterns, conducting the deep learning of large data sets across multiple verticals.
In testing, one of the biggest challenges will be normalizing the data that is flowing from multiple types of testing and understanding the structure of a test.
We will also be able to train an engine to conduct action based on a set of rules. If a test failed due to “X” let’s mark it as “Y” or alternatively do action “Z” in order to fix it.
AI and ML have big potential for test creation. Traditional test creation requires lot of time and highly-skilled automation engineers. But with AI, they can understand the structure of the page/app and accordingly identify all the objects associated with it with no need for coding.
If machines have enough notion about the technical structure of the service — as well as defined business logic — test creation can happen at the speed of light.
Test flakiness is one of the biggest problems of the testing industry. Even if you created a very good automated test, chances are pretty high it’ll fail soon. This is especially true if something changes in the UI, interrupting the object identification process (as some objects will change).
Organizations spend 15-25% of their time maintaining automated test cases. That is a significant amount of time that holds innovation (such as the creation of new test cases) back. The automated ability to observe these changes and adjust the automated test (self-healing) is key.
This is where the industry is at today. Test managers are sitting on large sets of data and asking themselves things like:
AI and ML can provide managers with more optimized and trustworthy testing. And the insights they receive from analytics can provide them with direction on what to do next.
So, the two areas that the combination of test automation and machines are looking at are:
Data becomes an enabler for valuable and successful continuous testing. Effective use of data should help with quick noise filtering (false negatives) and focus on actual failures that may resemble risk to the business.
The role of machines in the testing space grew in the last decade — mainly by replacing manual testing with test automation. Looking forward, it seems like human beings will remain close to testing, but we may see an interesting shift in the next decade from SDET (Software Development Engineer in Test) toward DSET (Data Scientists in Test).
This decade, smart test reporting and analytics will become more critical than ever before. Try Perfecto — with smart analytics, you’ll get the fast feedback you need to move at DevOps speed. Start testing with Perfecto today for free — and see how smart analytics can take your web and mobile app testing to the next level.
VP of Product Management, Perfecto
Tzvika Shahaf is the VP of Product Management at Perfecto. His experience includes business development, strategy, and investment in technology companies and venture capital firms. His passion is building new, powerful, and effective ways to collaborate with Global 2000 enterprises in order to resolve high-impact business problems using data-driven processes and analytics. Tzvika is partnering with leading DevOps teams to revolutionize the testing space by making it smarter, faster, and cost effective with a clear goal of maturing software delivery lifecycle. Tzvika is keynote speaker at industry leading events, blogger, and a Co-Author of the book, “Continuous Testing for DevOps Professionals: A Practical Guide from Industry Experts.”