Blog
January 16, 2020
Throughout the past decade, there have been huge advancements in the technologies used across the entire DevOps tool stack. Tools are enablers for productivity, and as such they need to serve their purpose, meet the needs of persona skillsets, and be flexible enough for changes.
In this article, we will focus on the most challenging tools segment within DevOps, and that’s the testing tools chain. Keep reading to learn why.
How 5 Types of Testing Tools Have Evolved (& What’s to Come)
When looking into the past decade of software testing tools, it’s imperative to understand what we refer to. It’s not just about the creation of the tests but also the test management, defect management, creation environment, test execution, and source control.
Let’s explore the following categories and realize the transformation such tools experienced over the past decade.
1. Bug Tracking Tools
Over the past decade, many defect reporting tools evolved and adjusted to the pace of software delivery maturity. From ALM tools like HP, IBM, Microsoft, and old-fashioned Excel spreadsheets, teams switched to more Agile-fashioned defect reporting and management tools. The leading ones include Jira, Monday.com, and Microsoft’s tools.
The key requirements for defect reporting tools today are:
- To be as close as possible to the Agile working environments.
- To be as open and integrated as possible.
- To work seamlessly with CI tools such as Jenkins and the likes.
That’s the main reason tools like Jira and Microsoft are so well adopted. They are close to the user stories and product requirements systems. They also work fine with Jenkins, TeamCity, etc.
What’s Next?
In the next decade, such tools will mature to be even more pluggable and integrated into the highly-adopted DevOps toolchain. This will make the submitting, tracking, and updating of defects as fast as possible.
2. Web Testing Tools
The changes in web technologies also impacted web testing tools.
Ten years ago, web app releases were relaxed with a slower release cadence of desktop browsers like Internet Explorer. DevOps teams have since transitioned to a much faster release cadence of apps and browser OS versions.
In the early 2010s, teams used Selenium and Selenium IDE along with other ALM legacy tools. These were used to slowly create and execute web application testing at low scale.
As the decade evolved, teams started embracing open source tools as well as those built on top of WebDriver protocol with JavaScript language. These were used for creation, together with headless browser tools led by Google Puppeteer. And toward the end of 2019, even modern codeless record and playback tools emerged on the market.
Looking at the web testing landscape now shows a change in the adoption of testing tools, including Puppeteer and Cypress.
In addition to the tools mentioned above, below is a deeper dive of solutions that expand the test coverage into other functional and non-functional areas of web testing.*
- Tools that provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber).
- Tools that provide assertion functions (Chai, Jasmine, Jest, Unexpected).
- Tools that can generate, display, and watch test results (Mocha, Jasmine, Jest, Karma).
- Tools that provide code coverage reports (Istanbul, Jest, Blanket).
- Tools that provide a browser or browser-like environment with control of scenario execution, UI testing, and more. (Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe).
- Tools that provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble).
What’s Next?
To get ready for the next decade of web testing, it’s important to constantly monitor the open source landscape. Teams must tune their tools to the current and upcoming web application changes. In most cases, they must mix and match the tools to their needs, skills, product coverage, and requirements.
Get your copy of our eBook, The Buyer's Guide to Web & Mobile Test Automation Tools
3. Mobile Testing Tools
Mobile has seen the most dramatic changes over the past decade.
In the 2010s, the market transitioned from Symbian and Java to Android and iOS. Mobile native apps have become much richer with advanced sensors, network operator abilities, and advanced APIs offer by Google and Apple.
All of these aspects led to the advanced mobile testing landscape of today, which is highly fragmented.
Teams must test the 4-5 leading Android OS families, as well as the 2-3 iOS families that are now running in parallel with iPadOS. Testing mobile native and hybrid apps has never been so complicated.
In addition, the capabilities of these apps are advancing. Facial sensors, 5G network connectivity, AR/VR capabilities, location and other context-aware capabilities, built-in AI engines, and foldable phones must all be included in today’s test scenarios.
Android and iOS developers need to stay on top of their feature’s functionalities and quality. Even without a code change, changes to the OS versions and new devices being released cause severe regressions to the apps.
For instance, Apple released over seven OS versions in a month for iOS 13. That means continuous testing must be fully employed to assess impact and address regressions caused by these dynamics.
Lastly, customer expectations are only rising. Users expect their mobile apps to perform flawlessly, fast, and with rich functionalities. These expectations are held regardless of platform, network, or location.
This means that developers and test engineers need to ensure that functionality is covered within regression and monitoring testing. They also need performance testing and other UX testing to be conducted.
Here are the main tools currently available to both developers and testers:
- Appium
- Espresso
- XCUITest
- Flutter
- Codeless (Apptest.ai)
Most of them are well-supported by leading cloud vendors, like Perfecto. Recently, initiatives like Apptest.ai that use codeless AI engines have grown and might evolve into a new set of functional end-to-end testing tools.
What’s Next?
Teams that are looking into the next few years from a mobile testing perspective should expect the continuous leadership of tools like Espresso, XCUITest, and Appium.
But teams should also start considering codeless tools that might enable lower-skilled developers to create test automation for mobile native apps.
In addition, as the mobile landscape continues to innovate and transform, foldable devices may become mainstream. Testing plans should be made accordingly as early as possible.
It’s likely to see late in 2020 an Apple foldable device that will be supported by iOS 14 in the same way that the Samsung Fold was released with Android 10 with during the fall of 2019.
4. CI Tools
The enablers for DevOps are continuous testing (CT) and continuous integration (CI). While there weren’t dramatic changes in the leading tools over the past decade, there were far-reaching changes in the level of importance and adoption of CI as part of the entire DevOps pipeline.
CI is now king in the way the pipeline is managed and orchestrated. And it is the gatekeeper for all DevOps contributors. Teams have allowed Jenkins to take first stage in the entire DevOps process. They have also integrated all their build-acceptance testing, deployments, and other scheduled tasks into CI servers. This expedites software delivery, provides visibility into each step of the process, and identifies risks.
One challenge of testing within CI is building highly stable and high-value test automation scenarios. This is one of the root causes of flaky builds, slow release cycles, and teams’ frustrations.
What’s Next?
Going forward, teams ought to be extremely rigorous about what enters the CI from a testing perspective. This can help them gain visibility into the efficiency of such testing and correct flakiness ASAP as not to carry poor CI debt.
5. IDEs
In the past decade, test creation has been the most challenging task of DevOps maturity. This is due to misaligned skillsets, tools, technology maturity, pace, and culture-related reasons. Teams were and are still challenged in getting continuous test automation right.
Test creation requires proper element identification (object IDs, XPATHs, etc.). It also needs a proper and up-to-date test environment (lab, test data, etc.).
The IDEs used over the past decade were able to serve as the canvas for test creation in development languages like Java, C#, Python, Ruby, etc. However, it wasn’t the responsibility of IDEs to cover for the inefficiencies in test automation authoring.
These IDES, with the exception of Eclipse, are the main ones still used by software test automation engineering (SDETs) and developers to create test code.
To compensate on the skills and practices in test creation, we have seen within these IDEs an advancement of frameworks that support BDD (behavior driven development) and TDD (test driven development).
Additionally, we have recently seen a rise of modern codeless and RPA (robotic process automation) IDEs that are based on record and playback of test automation.
A great win of the past decade has been cloud-based testing tools. It is almost a commodity in the market to leverage cloud-based vendors to access desktop browser VMs as well as mobile platforms (smartphones/tablets) to scale test automation, keep up with market changes, and ensure continuous security, data privacy, and backup.
Vendors like Perfecto are leaders in the space, and offer immediate access to the above digital platforms for testing, including tests from automation frameworks (Selenium, Appium, Espresso, XCUITest, and others).
What’s Next?
In the next few years, teams should expect an additional rise of codeless and RPA tools in parallel with a more advanced versions of IDEs. These will be more connected with build/CI/CD toolchains.
Back to topBottom Line
As we can see, the past decade brought immense change, challenges, and innovations to many different types of testing tools. Now with AI in software testing, there’s no telling what the future of software testing holds, or how we’ll continue to evolve DevOps throughout the 2020s.
With so much change on the horizon, you need a smart and reliable solution to support continuous testing. Try Perfecto, the world’s only end-to-end continuous testing platform.
From test creation to execution and analysis, plus an expansive cloud-based lab, all your testing needs are covered. Give it a try for free today.
*Classification of tools was created by Vitali Zaidman on Medium.com.
Back to top