February 14, 2023

How to Perform a Proof of Concept (POC) for a Continuous Testing Solution

Continuous Testing

Conducting a proof of concept (POC) in testing is an essential part of evaluating a test automation platform. With more solutions becoming available on the market, it is critical to know how to evaluate your future testing platform both quickly and comprehensively.

This blog will give an overview of the importance of the POC process, as well as a step-by-step breakdown of how to do a successful POC in testing. Feel free to read along or skip to the section that interests you most.

Table of Contents:

What is a POC in Testing?
Why Perform a POC for a Test Automation Solution
When to Perform a POC for a Test Automation Solution
How to Perform a POC in Testing

What is a POC in Testing?

A proof of concept (POC) in testing is the process of deciding whether your prospective test automation solution works effectively and can sufficiently help your team achieve its testing goals.

A POC is a critical step that an organization takes before investing in a test automation solution, whether they are switching from one platform to another or looking to transition from manual to automated testing.

This process should serve as an early indicator of how the solution’s technology fits your unique technical requirements, and should also be treated as a great way to validate the ROI of the solution in which you are planning to invest.

Why Perform a POC for a Test Automation Solution

Performing a POC for a test automation solution should both determine and confirm what type of automation your team wants to achieve in the first place. A POC can also assure your team that they are making the right decision when choosing a testing platform.

Before conducting a POC, it is worth considering what we refer to as “the three whys”:

Why conduct a POC for a test automation platform at all?
Why conduct this POC now?
Why this solution?

Let us take a closer look at each question more in-depth:

Why Conduct a POC in Testing at All?

While the first question may appear to be simpler to answer (e.g. you want to start automated testing or reduce sprint cycles), it uncovers a deeper question. Before starting a POC in testing, it is important to understand why you feel it is necessary to do test automation at all, or what has been stopping you from doing so beforehand with your existing toolchain.

In certain cases, a solution’s inability to automate critical tests for your organization, such as biometric authentication, image injection (QR, ID documents, face, etc.), or parallel test executions across dozens of mobile devices might have been the blockers for your test automation journey in the past. That being the case, it becomes much more important to drill down to the actual key use cases that matter most to you as part of your POC.

Why Conduct This POC Now & Why This Solution?

The second question will be harder to answer if the team has not adequately done their research (including internally within the organization) before starting a POC.

Starting a POC requires a lot of effort from within the organization. Most testing teams invest a significant amount of time to research, do due diligence, obtain security approvals, prepare environments with network teams, get any legal paperwork done, and more. These factors do not even consider the actual time spent on adjusting the test frameworks, test cases, the test engineers’ time, and so on.

In fact, it is estimated the cost of an average POC can cost an organization in the five-figure range, considering the number of people involved and those who support the project across departments.

Because of the resources involved from your organization just to execute a successful POC, it is vital that it is planned well. Most importantly, it is critical to clearly understand the licensing model of the solution beforehand, as well as have the potential size and amount of investment validated and approved by any procurement or finance function within your organization.

By going through these initial steps beforehand, you will not find yourselves in a situation where you have spent valuable time choosing the best solution that fits your needs most, only to be blocked by other factors that prevent you from having this solution put in place.

Nothing is worse than having a glimpse of the bright future, only to then face the harsh reality of being told that there is no way forward.

Once your team answers the first two questions, the remaining question of why this solution should be easily answered. Plus, you can rest assured that the POC is happening at the right stage of the test automation platform evaluation process.

When to Perform a POC for a Test Automation Solution

The POC should ideally take place as the final step of evaluating your prospective test automation solution. At this stage, you will have already done your initial research and gained a better understanding of what you are looking to achieve with an automation solution.

Before starting a POC in testing, you should already have done the following:

Obtain internal approval of your potential investment amount.
Align your team/s that will help facilitate the POC, as well as possible external resources such as contractors or partners.
Confirm your POC success criteria to all key stakeholders.
These insights will only help you make the most of your POC process.

How to Perform a POC in Testing

There are a few best practices to keep in mind to successfully conduct your POC in testing. At Perfecto, we take the following steps to ensure that our potential customers have a top-notch POC experience:

Define Success Criteria

The first step to performing a POC in testing is defining your success criteria. In this case, success criteria are two or three key test scenarios that you want to explore more closely with the continuous testing platform in question.

The test scenarios that you choose for your POC should either cover your most critical user journeys or address the most difficult test cases to automate. If you are looking to cover more advanced automation use cases that might require image injection, biometric authentication, or voice recognition, these should be incorporated into your POC success criteria.

At Perfecto, due to our years of experience in building automation across industries, we will always offer to build these key automation scenarios for you so that you can see the technologies working with your apps. Best of all, you will have a working project at the end of the POC, which you can both use immediately afterward as a quick-start project or showcase to your internal key stakeholders.

Of course, during the execution of the POC, your teams will have access to the solution themselves as well to carry out their own testing, with close guidance from Perfecto engineers.

By doing both a guided self-paced evaluation and a Perfecto-built automation project in parallel, you are guaranteed to have a successful and resource-efficient POC.

Determine Timeframe

At Perfecto, a testing POC will typically take anywhere from a few days to two weeks. Since you will gain the most by primarily focusing on your most critical test cases as part of your success criteria, your POC should typically not take longer than this.

That being the case, it becomes much easier to allocate a timeframe for the POC that works for all the teams who will need the solution.

Verify Your Expertise & Infrastructure Requirements

During your testing POC, your team can verify your expertise requirements in two ways. First, you should determine the experience level of those on your team who will need to use the solution.

For example, if the critical test case that you cover in your POC includes scriptless test creation, your expertise requirements will not be particularly high. But if your critical test cases cover a more complex user journey, the people involved in the POC should have relevant experience to delve into that user journey in depth.

Evaluate the team behind your testing solution

The second way to verify your expertise requirements is by evaluating the team behind the test automation platform that you are evaluating. While a solution may offer the technology that you require, you are ultimately working with a vendor team to make the most out of that technology. During a POC in testing, it is recommended to take note of the expertise that the vendor team can provide. Raise support tickets during your POC and note how much the vendor team can help guide you toward fulfilling your automation goals.

For infrastructure requirements, it is also important to verify whether you will need to use a VPN or private infrastructure to conduct your POC in testing. These requirements are especially relevant if you have an insurance or financial services app where you are dealing with more sensitive information.

Choose Your Testing Devices

Another critical factor to consider when doing a POC in testing is choosing which devices to test on as part of your evaluation. The best way to do this is by running your POC with a test automation platform that offers a comprehensive device lab.

Choosing a platform with a device lab is particularly important to make sure the solution you are going to use will guarantee day-one support of new OS versions, betas, and mobile devices. The ability to test on the latest and greatest devices is critical, especially if your application is customer-facing.

For example, iOS 16 had a 10.45% adoption rate within the first three days of going live. Therefore, if your customers are using your apps within these three days and you cannot test them from day one (if not before), you risk a bad user experience for 10% of your customers due to undiscovered issues on this new OS platform.

If your testing platform in question either does not have a device lab or if you have extenuating circumstances around the devices you would like to use, you should also evaluate the vendor team accordingly to see how they take care of that as part of their preparations for your POC.

Embrace the Echo Back

Finally, after your POC is over, you should create a detailed echo back report to summarize what happened during the POC and whether the vendor was able to deliver on the success criteria promised at the beginning of the POC process.

Your provider should work with you to build this echo back so that you are as equipped as possible to present the results of the POC to upper-level management. Plus, the vendor you work with can then demonstrate that they clearly understand your company objectives, the challenges you want to overcome, and the key technologies required for success.

Ultimately, the echo back report should be a strong indication of a future trusted partnership between you and the vendor. Achieving true continuous testing is no small feat, and you need to feel confident in the team that will help you implement this test automation solution in the future.

Other points that should be included in your echo back include:

Challenges of the current state of your testing.
The negative business impact of maintaining the current state of testing.
A high-level summary of your POC success criteria and whether they were achieved. In other words, was the POC successful?
The positive benefits that the solution provides, with detailed examples from the POC.
The key technologies and use cases validated in the POC and the results.
The ROI of the solution you are planning to put in place, compared with the business impact of maintaining the status quo.

Bottom Line

While the process discussed in this blog is often referred to as a proof of concept (POC) in testing, you can also consider this final stage of evaluation as a Proof of Value (POV). Because ultimately what you are looking for with a new continuous testing solution is to bring value via the use of technology.

At this important stage of exploring a testing platform, the POC in testing should be the way to show that the solution in question can deliver what was promised and how much it will bring value to your business, as well as reduce negative business impacts.

At Perfecto, we pride ourselves on working together with our customers as a team to define where they want to go in their automation journey. Our customer base, which includes over half of the Fortune 500, continuously relies on our 15 years of industry experience to lead them to test automation success.

Explore how Perfecto can become your partner in test automation by setting up a 1:1 demo with our experts.

Get Custom Demo

The 2023 State of Test Automation