Blog
January 10, 2019
Enterprises are under constant pressure to improve software delivery cycles while maintaining — or even improving — quality. The only feasible way to iterate more quickly while ensuring quality is to implement continuous testing as part of the overall development strategy.
Continuous testing (CT) allows organizations to gather fast feedback per each code change (commit) as part of the continuous integration (CI) workflow. When implemented properly, CT can help organizations attain true DevOps status. The only real downside associated with CT is the vast amount of test results data that is generated with a high number of commits - and subsequent volume of test cases - throughout code cycles.
Continuous Testing Produces a Lot of Test Results Data
Testing is a critical component in building a successful DevOps organization. The more tests executions you have across a large array of platforms (mobile devices, tablets, desktop browsers), the more test results data is created. Analyzing, understanding, and filtering the results data quickly is critical in order to prevent bottlenecking the DevOps pipeline.
In a DevOps environment, the window to review and qualify test results has shrunk dramatically from days to hours (and even minutes in some cases). What’s no longer acceptable is a situation where a regression testing suite is running for 2-3 days and requires a similar amount of time to analyze results.
Our research at Perfecto tells us that organizations spend between 50-72 hours per regression cycle analyzing test results, filtering out noise and assessing failures which may impact their software releases. It is essential that teams have a better solution for test reporting in order to analyze data, triage issues, and act upon failures with the best possible insights. Unlike desktop browser testing, mobile devices create more complex issues as failures can be caused by network connectivity, device locks, improper object use, popups and more.
The Essential Toolbox for DevOps Analytics
When dealing with large test results datasets in a CI/DevOps environment, there are a few essential tools that teams need:
- Executive dashboards that include quality heatmaps, continuous integration (CI) dashboards. In the screenshot above, users such as dev managers and QA managers can easily examine the pipeline, see CI trends related to time, build health and more. Upon finding an anomaly in the CI pipeline, managers and practitioners can easily drill down into the single test report and into the issue.
- Single test report visibility with advanced reporting artifacts (Videos, screenshots, logs, access to Jira and Git, etc.)
In the single test report, practitioners have access to the entire flow of test steps, video, logs, screenshots as well as access to Jira - even the ability to drill down into the source code to easily locate the issue. Tools like Xray for Jira and Perfecto integration make it easier to track manual and automated test executions in Jira, streamlining the process of reviewing test outcomes and assigning issues.
- Cross-platform reports show the UI/UX simultaneously across multiple screen sizes, resolutions, and platforms. Since the boundaries between the digital platforms need to be seamless, whether on desktop browsers or mobile devices, having the ability to see the UI/UX on multiple form factors (and layouts) at the same time is essential to assess overall quality.
- Noise reduction tools powered by AI and analytics allow teams to filter out issues that are caused due to various issues that are NOT software defects – things like device connectivity issues, wrong scripting (bad objects, timeout issues, popup handling, etc.). Finding which error classifications are true application bugs is critical to efficiency. Having a tool with the ability to locate each failed test execution, the root cause, and the classification category is a huge productivity boost for both the test engineer and the developer. Things like popup handling, “element not found”, device connectivity etc., are among the top issues that slow down defect resolution activities.
- Actionable insights with prescriptive ways to resolve issues. Once an issue is detected, classified and reported, having deep insight (see screenshot) brings practitioners halfway towards the resolution of the issue. In the example above, the error is due to a popup.
Bottom Line
To excel in software delivery for mobile, native, or responsive web apps, teams not only need better automation as part of their processes, test flows, and CI/CD workflows, but also a test results analytics platform to manage all of their test results data in a way such that the teams can evaluate the data, act upon it, and deliver iterations and features with confidence - and quickly!
Since DevOps involves a collection of team members from all parts of the SDLC process, the central platform needs to meet the needs of ALL team members.
That’s why executives are keenly interested in high-level, quality dashboards offering CI health (pipeline) status, while developers and testers care most about test artifacts, noise reduction, and single test reports.
To learn more about next-gen reporting and analytics for DevOps, visit our Smart Reporting and Analytics page. Or, give it a try to see for yourself.