Continuous Quality Black Belt Program

Open Black Belt Program configuration options

A Perfecto customer — let’s call them Customer X — had a mobile banking application which was a key element of their digital strategy. When Perfecto began working with their quality assurance (QA) team, their software development lifecycle (SDLC) was strictly a waterfall process from requirements gathering through release. Customer X wanted to move toward a hybrid/Agile SDLC utilizing continuous integration (CI) with automated build verification and regression testing.

The costs associated with automated testing did not produce the results that the management team sought. The automated regression suite of 2000 test cases running on three mobile platforms took two full weeks to execute. The QA team shifted from HP Unified Functional Testing (UFT) to Java Selenium for their mobile automation, but still did not achieve their desired outcome. The end result was a heavy dependence on a large team of manual testers to ensure mobile application quality — 4000 test cases per release.

Enter Perfecto Professional Services.


Black Belt Program

The first task of the Perfecto Black Belt was to analyze the customer’s digital strategy objectives, SDLC, and evaluate the automated test strategy and implementation. There were many opportunities for optimization, including:

The automated testing framework

  • No parallel execution of tests, which diminished the full value of test lab hardware.
  • No method for collecting and storing valuable execution data for visibility into the application’s overall health.

Scripts were written without best practices

  • Very slow execution due to a lack of dynamic script synchronization.
  • Heavy maintenance and slow development due to low modularity and code reuse.
  • Limited testing functionality due to use of proprietary mobile driver instead of open source Selenium Remote Web Driver.

Lack of automation in software build process

  • No CI processes, preventing a true Agile environment and slowing the entire development cycle.
  • A high number of critical defects escaping into the full regression testing cycle due to low coverage.

The Solution

From August to December, many issues that the automation team faced were addressed through management and technical sessions led by the Perfecto Black Belt.

  1. The first and primary task was to implement a mature open source automated testing framework.
  2. The second task, completed in parallel, was to walk through the automated test scripts and identify areas of optimization through Perfecto best practices.
  3. Also in parallel, the automated and manual test cases were evaluated to optimize test coverage.
  4. The third task was to draw the QA and development teams into closer cooperation to ensure the benefits of automated testing were realized earlier in the SDLC.
  5. Fourth, the Perfecto Black Belt mentored the automation team to help them design and build more efficient and robust automation scripts.

The Outcome

Within five months of initiating the Perfecto Black Belt Program, several game-changing achievements had been realized by Customer X’s mobile application quality assurance team:

  1. Automated test execution time was reduced by 75% through code review and optimization.
  2. The fully automated regression suite of 2000 test cases was reduced to 1500 while increasing test coverage, resulting in lower cost of script maintenance.
  3. Full parallel execution reduced automated regression test time from two weeks to seven hours using five threads per platform (15 simultaneous tests).
  4. Implementation of CI through Jenkins automated the daily execution of unit testing, build verification testing, smoke testing, and regression testing.
  5. QA now maintains ~70% automated coverage release to release for regression.
  6. The increased automation coverage has freed up the QA team to focus on expanding coverage for new features and automated unit testing for the development team.
  7. The Perfecto Black Belt implemented a data warehouse using MySQL and each execution is now committed to a database to allow aggregation of test execution results in real time and comparison across specific runs or software releases.

Summary

Customer X achieved an industry-leading digital strategy with the velocity and quality unachievable without an automation-enabled Agile SDLC. As Customer X understands, the world is a rapidly changing place and companies who cannot compete in the digital arena will not survive.

Does your automation team have the experience required to meet this challenge? If not, Perfecto can help.