Blog
January 8, 2021
Let me be frank.
Testing today is hard. Truly.
Don’t let anyone tell you differently.
There are so many variables that complicate quality: the plethora of mobile and web permutations, the growing number of frameworks (open-source and commercial), the global spread of people and resources, the various skillsets of team members, the complexity of the test environments, and more.
How can a team ensure that the businesses most-essential assets, their web and mobile apps, are performing as they should?
Without working 80+ hours a week?
Every testing solution you find online will promise to help you achieve this, deliver quality at speed.
What’s different about Perfecto?
I get this question a lot. I’ve been working with Perfecto for nearly a decade. I can honestly say that we have — through our years in the industry — developed a market-driven platform that brings real value to our customers.
(It is why over half of the Fortune 500, when they need to make testing work, have come to us for help!)
So, in this blog, I will share how Perfecto customers:
- Expand test coverage without skyrocketing costs and stress.
- Scale automation with what I believe is the lowest cost per execution.
- Democratize quality by putting useful tech into the whole team’s hands.
- Become more efficient with Perfecto’s AI-enhanced analytics.
Will what has worked for them, work for you? You may be skeptical now, but I assure you that you needn’t be.
Test on 100s of Devices for the Price of One Smartphone
Some teams believe they can reduce cost by assembling their own device lab.
If you are considering a DIY lab, I urge you to break down everything associated with such a goal. Many who go the DIY route end up miscalculating the time and investment necessary.
Let us start simple for arguments’ sake.
If you are going to set up a lab, you will need to buy at least one device. The base cost of a modest smartphone at $650 USD. (Though many devices cost closer to $1,000).
Add device hosting, the carrier contract, accessories that connect it to the organization servers, and ongoing maintenance and the cost hits $1,250 per year. Bear in mind that this device is not easily shared across the organization.
Compare $1,250 per year for one device DIY to the $996 per year it costs to access the Perfecto public cloud for manual testing.
Plus, with Perfecto you can access a variety of devices — not just one.
To me, this is a simple choice.
Need Private Devices?
If you need your own devices or already have a lab of your own, Perfecto's private cloud also delivers substantial savings over a DIY lab, particularly in the form of labor costs. Managing a lab is expensive and difficult and includes ongoing costs that are often missed, including:
- Procuring devices, smartphones, and tablets (IT labor).
- Adding new models when they come out (IT/R&D labor).
- Maintaining a lab for local and remote users 24/7 (IT labor).
- Updating lab DUTs with new OS versions, ongoing (IT labor).
- Hosting devices with servers, network connections (IT labor).
It adds up — and quickly. With Perfecto private cloud, we take care of all of it. Everything is always on, updated, and available.
Bottom Line
Perfecto’s cloud-based lab is much less expensive than a DIY lab, which is difficult and costly to maintain. With instant access to the latest devices, you can quickly expand coverage.
Most importantly, Perfecto also comes with rich features: network virtualization, maximum security, global availability, carrier network, location mimicking, object spy as a Segway for automated scripts, and more, that give you an upper hand on quality.
Execute an Automated Test for Less Than a Dollar
Continuous testing is literally continuous.
You exercise old and new test scenarios on platforms daily. Sometimes a few times daily. With such a constant cadence, the cost per execution becomes a huge factor in the overall DevOps budget containment.
Like in the lab cost considerations above, executions come with many factors to figure into the overall cost: size of test suite, # of environments, cadence, device/OS permutations, # of applications, etc.
At Perfecto, we are helping organizations become as lean as possible. In fact, one of our customers recently assessed the overall costs for executing 250,000 tests weekly.
With Perfecto, they reduced the cost of a single test execution from around $9.00 to $0.91. They were able to use the savings to test more scenarios and expand coverage. To my knowledge, there is no other combination of technology and people bringing this much value to testing at scale.
Bottom Line
At Perfecto, we know how to help organizations large and small scale their automation, sensibly. The ROI of automation with Perfecto goes further than the low cost of test execution. Consider the value of being able to reallocate budget dollars to increase the level of testing, expand coverage, or make other investments.
Elevate Testing Without Turning It Upside Down
Change is hard. It is natural to encounter resistance when changing how you do things when investing in a cloud-based testing platform like Perfecto.
This resistance just does not happen with Perfecto. Why? Because it is agnostic to skills and technology.
Meaning: It is not a straightjacket that your organization must squeeze into. It is the opposite. Perfecto is an open and integrated platform that allows for existing ways of working, while opening a few new doors for efficiency and collaboration.
Perfecto bridges the gap between today’s myriad of tools, people, processes, and goals.
- Developers create automated scripts with Espresso, XCUITest or Cypress,
- Business testers do manual testing AND automate using Quantum BDD or codeless tools.
- SDETs create and execute their Selenium/Appium scripts in parallel in the cloud.
They all can work in the same platform, giving the organization a single pane of visibility into all quality activities.
Bottom Line
Bringing everyone onto one solution like Perfecto enhances individual productivity while making collaboration more fruitful. It is challenging to attach a concrete dollar value to this point, but based on conversations I have with customers, it is one of the big difference makers. It allows for maximum test coverage and higher confidence in quality of deliverables.
At a Glance, Know What, Where, & Why Tests Fail
The biggest ROI from test automation (or any other type of testing) comes after the execution phase. It arrives with the knowledge that a test either passed or failed and needs to be fixed.
When reading a common pass/fail report, deciphering what tests broke and why is an intensive process.
With Perfecto, teams can squeeze as much value from testing as possible by getting immediate feedback on test results. You have access to things like:
- Rich media artifacts (logs, videos, screenshots, HAR files).
- Root cause analysis classification.
- CI dashboard.
- Heatmaps and other reporting features.
Going even further, Perfecto's ML-powered detection and classification of errors separates false negatives from real failures for quieter testing. You only have to digest what's important.
All of which gives you different lenses for understanding — efficiently — what, where, and why test are failing.
Bottom Line
We have not clocked any of our customers on the time they’ve saved with our comprehensive test analytics. Nor have we asked them to disclose the dollars they’ve saved by making testing more pointed and efficient. But we do know that the reporting and analytics in Perfecto, which have been honed over the years with constant feedback from our customers, is central to their understanding of ROI.
Bringing It All Together
Mediocre quality, escaped defects, and outages have never had greater business implications than they do in today’s highly competitive digital marketplace. At the same time, teams are releasing faster than ever: daily, weekly, monthly.
I hope this post has illustrated a few main reasons why our customers value logging into Perfecto to keep their apps working well and their businesses working efficiently.
Deciding on the platform that fits best involves many considerations, trade-offs, priorities. To make the right decision, quantified labor and material as much as possible while keeping in mind the tangible and intangible points highlighted here.
See How Perfecto Delivers
If you want to see more specifically how customers use the Perfecto platform, reach out for a 1:1 custom demo.