August 7, 2019

TestCraft vs. Record-and-Playback Tools

Scriptless Testing

Test automation and test recording (also known as “record-and-playback”) are two different QA testing methods that are often confused. Both define themselves as “automated” and “codeless,” and while this may be the case for both, this idea of codeless automation manifests itself differently with each type of method. Each tool comes with its own set of advantages when dealing with your software development operations.

In this post, we’ll set the record straight and highlight the differences between test recording tools and our own codeless test automation tool − TestCraft. This comparison will be broken down into three categories:

  • Test creation.
  • Test execution.
  • Test maintenance and updating.
Test CreationDual creation mechanism:
- Drag & drop interface
- Live point & click modeling

Create test flows prior to code — Shift Left benefit
Single recording mechanism

Record only after the application is working
Test ExecutionOn-the-Fly binding (during runtime)

Run the tests on multiple browsers simultaneously

Monitoring capabilities
Playback mode
Test Maintenance & UpdatingReuse elements and flows across scenarios

On-the-fly fixing

Dual output — data & scenario

Single output - a recording


Test Creation

With recorders, the test creation process is super simple – capturing user actions in real-time. The benefits are clear; it’s fast and easy to use. Tests can be created in the span of time it takes to perform the action. Yet it’s important to note that if a mistake was recorded, there’s no way to fix it. You need to go back to square one and record the whole scenario again.

TestCraft cannot compete with the speed of a recorder, but can compensate with two innovative test creation mechanisms:

  1. Creating visual flows by dragging and dropping elements, or sequences of elements, onto a virtual canvas.
  2. Modeling flows by pointing and clicking on real, online elements in the web application, during runtime.*

*Though the second mechanism might sound like a recorder, there’s a big difference. We’ll explain this further when we discuss test maintenance and modification.

Create Tests Before or During Coding With TestCraft

A major implication of the above differences is that with TestCraft, you can create tests even before the actual code has been written.

Working on the virtual canvas, you can design the tests in an abstract way based solely on the product wireframe or product flow (this can be done by the product manager or the tester, in case he’s part of the product design process). Later on, after the development team has coded the feature, you can get the specific details of each flow and bind the actual elements during runtime. This has a strong shift-left benefit for companies that pursue continuous testing to achieve an agile work process.

With recorders, you do not have this advantage, as you have to wait for the feature to be completed in order to create the recording.

Dynamic Data Injection With TestCraft

Another major difference between TestCraft and recorders is that with TestCraft, you can receive data from backend components to be integrated into the test flow and use it for validation purposes.

Let’s look at a real example from one of our customers – a bank – for dynamically injecting data from an external source and how this feature can come in handy. This bank often tests live, on the actual site, to test a process that involves a confiscated account. Yet the identities of confiscated accounts change daily; some accounts are being settled while others are constantly being added.

TestCraft knows how to send a query to the bank’s database before every run, retrieve confiscated account details, and inject this data into the test model.

As far as we can tell, recorders don’t offer this functionality.

Test Execution

No Reruns With TestCraft

With record-and-playback tools, the execution stage is very straightforward; whatever was recorded is being “played back” to test the usability of the web application. If the test breaks, you would have to re-record it and let it run all over again.

TestCraft, on the other hand, gives a more hands-on execution of tests called “on-the-fly rebinding”. If something breaks, you can fix the issue right then and there by rebinding the elements. You can modify your tests in real-time without having to worry about re-recording the entire test scenario from scratch.

Our execution mode alerts you when an element breaks so you could fix it on the spot by pointing to the right element. TestCraft will bind the element and continue the test execution. A second execution mode runs completely automatically and will fail immediately when it identifies any breakage.

Monitoring Implementation With TestCraft

Another execution implementation of TestCraft is for monitoring, not just testing. High traffic sites, such as e-commerce or online services, often deal with dozens, if not hundreds, of processes simultaneously. Even the smallest failure can have direct repercussions on sales or user satisfaction.

With TestCraft, you can create flows easily to be used for application monitoring. After creating one or more flows, you can schedule them to run on any interval you choose. This ensures that once it identifies an anomaly in performance, you’ll know about it almost instantly.

TestCraft also provides detailed reports for every execution, from the single element to comprehensive performance monitoring reports.

TestCraft Screen


Test Maintenance & Updating


Self-Healing AI With TestCraft

Another key feature that TestCraft has over record-and-playback tools is a smart element locator. Using artificial intelligence and machine learning, TestCraft can overcome changes in your application and update your test flow automatically. This self-healing mechanism makes tests more resilient, reducing the time and resources needed to do proper test maintenance.

Reuse Elements and Flows With TestCraft

Test automation delivers speed and ease of use. Record-and-playback tools definitely offer both; they’re super easy to use and also pretty quick to operate during the creation stage. But what happens afterward, when processes need to be edited or adjusted to accommodate code changes? TestCraft offers the same ease and speed of the creation stage during any following development.

Let’s say you need to test a purchasing process in an online store. There are three main components to this:

  1. Arriving at the site
  2. Selecting an item
  3. Checking out

With a recorder, you will go through the entire process as a user. With TestCraft you can either drag-and-drop elements onto the canvas, or model through point-and-click on live elements in the shop itself, during runtime.

The result will be similar on the surface – a documented scenario for a purchasing process.

But what happens if you want to test the purchase of two products? With a recorder, it is "back to the drawing board;" you will need to record the whole scenario from the beginning.

With TestCraft, it is much easier. You can simply duplicate the second component and run a new test for the purchase of two products.

You can reuse components across various flows with higher ease of use.


TestCraft Reuse


Dual Output – Data & Scenario With TestCraft

All of the above (and much more) is due to the fact the TestCraft produces two independent outputs – data and scenario. In contrast, record-and-playback tools only have a single output – a recording. The ability to edit, update and implement changes in each of the outputs separately is where TestCraft’s strength shines through.

TestCraft provides fast, easy, and complete control over testing and the ability to integrate the QA testing operation deep into the agile development process. Deployment of features and product cycles becomes much smoother and faster – allowing for a more thorough software testing process and improving the overall quality of your web application.

Witness the Difference With TestCraft 

The final element that distinguishes our product from test recorders is that TestCraft is doubly pure: it is based on pure SaaS – no agents or extensions and is purely codeless – no tech skills required.

Watch TestCraft Demo