what is test automation
September 16, 2020

What Is Test Automation?

Automation
Continuous Testing

By

Test automation is a popular approach to testing. In this blog, we cover everything you need to know about test automation — otherwise known as automated testing.

Keep reading or jump ahead to the section that interests you most:

What Is Automated Testing?

Automated testing is a method in software testing that leverages automation tools to control the execution of tests instead of a human tester. It then compares actual test results with predicted or expected results. Automated testing offers greater efficiency and faster time-to-market for your projects.

Automated testing is also called test automation or automated QA testing. When executed well, it relieves much of the manual requirements of the testing lifecycle. 

Types of Automated Tests

Most tests done manually can be automated. What a user will manually perform can be replicated with an automation script. However, not all tests should be automated and we’ll look at this later in this article. 

Here is a safe list of test types that can be automated without a doubt.

1. Unit Testing

Unit testing is when you isolate a single unit of your application from the rest of the software and test its behavior. These tests don’t depend on external APIs, databases, or anything else. 

If you have a function on which you want to perform a unit test and that function uses some external library or even another unit from the same app, then these resources will be mocked.

The main purpose of unit testing is to see how each component of your application will work, without being impacted by anything else. Unit testing is performed during the development phase, is considered as the first level of testing.

2. Integration Testing

In integration testing, you test how the units are integrated logically and how they work as a group.

The main purpose of integration testing is to verify how the modules communicate and behave together and to evaluate the compliance of a system.

3. Smoke Testing

Smoke testing is performed to examine whether the system build is stable or not. In short, its purpose is to examine if the main functionalities work properly so that testers can proceed with further testing.

4. Regression Testing

Regression testing checks that a recent change in code doesn’t affect any existing features of the app in question. In simple terms, it verifies that changes made to the system did not break any functionality that was working correctly prior to their implementation.

To conclude this section, there are several types of tests that can be automated. Test automation is when you configure a script/program to do the same steps as you would do to manually test the software. 

You design that script to wait for certain values to be returned or certain events to take place.

In the end, the script will perform whatever you instructed it to and it will show you if the test result is the same as the one that you expected. 

Automated Testing Example

Let’s apply the description above to the following code.

The code above is a simple function that will return the current time of the day as a string: “Night,” ”Morning,” and “Afternoon.”

To manually test this function, all we’ll have to do is to run this program and somewhere inside it to call the function that we are interested in. Therefore, I updated this code snippet with the following statements, that are placed after function getTimeOfTheDay():

console.log(newDate().getHours());
console.log(getTimeOfTheDay());

If I execute this code now, the output will be the following:

I executed this program at 17:44 (5:44 PM) so the current hour is 17 and the returned value from the function that I want to test is Afternoon. So until now, my function is working as expected. 

If I want to transform this test and automate then I can write the following script.

With this in place, I no longer need to manually call getTimeOfTheDay()function in my code. I can simply run this automation script which will run the function and inform me if the returned value is Afternoon

Now this test will pass only if we run this script between the hours [12,18)-[12 PM, 6 PM). 

Otherwise, it will look something like this:

This was a trivial example. A real case scenario would be a test script used to automate a regression suite. That’s a script that will test if the application still works as expected after minor/major code changes.

Why Test Automation Is Important

Testing code only at the end of a development cycle simply doesn’t cut it anymore. That's why test automation is important. 

Agile and DevOps are the new models for modern software engineering. This has changed the way code is being developed, tested, and used by businesses and consumers. And almost every organization today offers some type of software (e.g., mobile banking app or an online flight check-in tool).

This is especially true when it comes to testing.

New software releases are being delivered faster and more regularly than ever. It’s become crucial for companies with any level of digital footprint to improve the efficiency and accuracy of their testing to keep up with their competitors.

Testing at every stage of the delivery pipeline — known as continuous testing — is now an integral part of the way companies operate today. And test automation is key for your success.

[Related Blog: 7 Test Automation Benefits]

At the same time, test automation can save a lot of time and, ultimately, a lot of money. Manually going through the same scenarios each time a new software change occurs just to ensure that the other features have not been damaged can be a tedious process that will only take longer and longer.

 

The Most Common Challenges of Test Automation

However, test automation does have some challenges, especially when it is first adopted. 

•    Complex test creation leads to a high percentage of manual testing.
•    Reporting noise like false negatives results in irrelevant test feedback.
•    Untrusted test automation leads to a high percentage of manual testing.
•    Execute parallel testing while scaling requires significant IT investment.
•    Managing an in-house lab is costly and difficult to keep up to date.

These common challenges are easily overcome with the right automation strategy and the right automation tools.

[Read Our Guide: Best Practices to Solving the Top Digital Test-Automation-At-Scale Challenges.]

What About Manual Testing?

Test automation doesn't mean the end of manual testing. 

Rather than replacing human intuition and problem solving, test automation is about automating the right tests. Not everything that can be automated should be automated. As a quick example, if you have something that you need to test ASAP and it’s just a one-time job, then there is no need to waste time in writing a script to do it for you.

Automating the right tests will help your team by giving developers and testers back valuable time. They can then focus on more important tasks. For instance, they can create new features, solve tricky bugs, and deliver higher-quality software.

[Related Blog: Automated Testing vs. Manual Testing]

Barriers to Automated Testing

While automated testing today is the preferred testing method, organizations face barriers when trying to implement automation.

Creating automation scripts is a significant barrier.

Businesses struggle to step from basic automation to enterprise-level automation. Testing environments must be stable and reliable enough to support automated QA testing at scale. Because test automation executes tests faster and more often, teams must also have tools to manage the significant volume of data that is created. 

As an example here, automating a UI test and having it executed against multiple browsers and OS can be a complicated process that will require some experience. 

Scaling test automation simplifies the entire testing process.

It makes it easier to develop and organize test scripts and environments, as well as to track and share results across teams. And most automation test tools actually help with important tasks like creating test data and setting up test environments.

How to Choose Which Tests to Automate

The truth is that many tests can be safely automated. Think of all those common, highly repeatable tests that make even the savviest testers out there want to gouge their eyes out by the end of the day.

Most unit, integration, and performance testing can be easily automated — developers and testers only need to intervene when the results don’t match up to expectations.

Tests that are a perfect fit for automation share a few basic characteristics:

High Volume and Repeatability

It’s a waste to create an automated test script that only needs to be run once.

Test automation is designed for running time-consuming test scripts that need to be repeated over and over again. 

Testing different OS/browser combinations or high-volume batch testing overnight are solid automation choices.

Determinant Outcomes

Test automation works best when tests are both repeatable and have determinant outcomes.

That is, the results should have fairly predictable outcomes that a test script is 100% capable of catching. Stress and load tests fit nicely into this category.

Business-Critical Tests

For tests that can cause an interruption in service and potentially damage one’s business, test automation can help ensure new features don’t break existing ones.

Smoke tests, sanity tests, and regression tests are good candidates for automation — especially when they need to be tested across every version and release of an application.

Test Data and Environments

Remember, test automation is more than just automating tests. It can also be used for automating tasks like setting up test data and environments. Some test automation tools out there can even build test scripts before code is written — simply by defining the desired functionality first.

Unopinionated

You cannot automate matters of opinion. No matter what tool you use, it’s impossible to determine the user behavior and his impression about your software.

This is where usability testing, Beta testing, A/B testing, and so forth is still a hands on activity.

We all know that getting some feedback from your users is important, but it just can’t be automated. 

Automation Testing Tutorial: Step-by-Step

No matter what test automation tool you’re using, the same basic process applies. Here's a quick tutorial of how automation testing works. 

1. Prepare the Test Environment

Establishing a base state for the test environment is the first step. This typically involves creating realistic test data and setting up one or more environments. During testing, data will be manipulated and/or the system will be put into a specific state. Afterward, there has to be a base state the system can return to.

If we go back to the example before with that getTimeOfTheDay() function, we said that the code that we wrote will only work if we run our test suite between the hours [12,18).

To fully comply with this first step that we defined, then we need to prepare our environment and configure our test not to take into consideration the real date, but rather we should be able to set a custom date that we want. Only in this way we can say that we fully automated that test.

As a quick best practice tip, tests should be done in all corner cases. Therefore, in our situation, we should test the function for hours 0, 5:59, 6:00, 11:59, 12:00, 16:59, 17:00 and 23:59

The refactored test script to match these requirements would be the following:

2. Run the Test Script

Tests run according to specifications set out in the test script. Automated testing software manages and tracks every test across all environments. Test automation tools run tests accurately and tirelessly — day or night — without any risk of error caused by human fatigue.

If we run the refactored version of our test suite, the result would be the following:

3. Analyze the Results

Once the testing software has finished sniffing out every bug, results are highlighted as either a pass or a fail. If there was any problem during the test (or the results were inconclusive), developers can review a complete log to find the problem in the test script and/or manually rerun the scenario.

This was a brief introduction with some quick steps that you can use right away in order to automate your tests. However, there is more to know, if one might want to learn.

Who Uses Test Automation?

In DevOps and testing there are three personas to be familiar with. Depending on the persona, there will be different technology and methods required.

The skills that vary between each persona require creation of test automation in multiple methods like low/no-code, codeless, BDD, and code-based.

Business Tester

A business tester is a stakeholder from the business team.

Almost always this type of tester has less technical skill and the testing they implement is manual and exploratory. Business testers are valuable because of the human and contextual knowledge they bring. The business stakeholder is often responsible for codeless testing.

SDET (Software Developer in Testing)

A software developer in testing has significant code knowledge and experience in testing.

They can create functional and nonfunctional code-based test automation scripts with tools like Selenium and Appium, among others. The SDET is always accountable for the code-based testing.

Software Developer

The software developer tester creates unit and build acceptance tests.

Software developers also operate in code-based testing. They also work in UI and UX tests, which are manual.

Test Automation Tools

As we previously mentioned, you need specific tools to automate a manual test. A special tool is required. This tool can be a simple testing framework like Jest, a special software framework like Selenium, or even an entire platform that allows you to basically do everything you need.

Not All Test Automation Tools Are Equal

It is important to note that not all test automation software is created equal. The above process outlines the best practices and best-case scenarios for test automation.

However, some automation software vendors do not offer test reporting features. Other solutions don’t have robust test script creation tools. When selecting your test automation solution, be clear about the essential features required for optimal test automation. Otherwise, you could put yourself at risk for a number of testing challenges.

Furthermore, some apps are more complicated to test than others. Mobile apps can be more or less complicated than web applications. Industries like healthcare and automotive often have specific and regulated requirements, which creates complicated test conditions.

For success, the automation solution must match both the application and industry. 

And lastly, the team experience must be taken into consideration when choosing an automation tool. If your team doesn’t have that much code experience, then a codeless solution would be better and more profitable rather than choosing to put things on hold and do some trainings for your team.

Perfecto's test automation solution does all of that.

Try Test Automation Tools Today

Perfecto’s test automation solution offers key features that build high-coverage test suites, including:

  • Accessibility.
  • Non-functional performance testing.
  • Usability.
  • Unit testing.

Test automation is a perfect solution for common, repetitive, and high-volume testing. Coordinating and managing testing are now becoming much easier with Perfecto. You can track and share testing results from a single, centralized location.

This gives you more thorough test coverage, because more testing can be accomplished. While there is definitely manual work still involved in testing, using Perfecto improves the accuracy and coverage of testing for teams competing in an increasingly fast-paced software market.

To learn more about how Perfecto can simplify and improve all types of testing at every stage of the development cycle, start your free trial today.

Try Perfecto

 

Related Content

This blog was originally published in September 2019. It has since been updated for accuracy and comprehensiveness.

Eran Kinsbruner

Eran Kinsbruner

DevOps Chief Evangelist & Sr. Director at Perforce Software, Perfecto

Eran Kinsbruner is the author of the 2016 Amazon bestseller, “The Digital Quality Handbook”, “Continuous Testing for DevOps Professionals”, and “Accelerating Software Quality – ML and AI in the Age of DevOps”, which was named the “Best New Software Testing Book” by Book Authority. He is Chief Evangelist and Senior Director at Perforce Software. With a background of over 20 years’ experience in development and testing at companies such as Sun Microsystems, Neustar, Texas Instruments, General Electric, and more, Eran holds various industry certifications such as ISTQB, CMMI, and others. Eran is a recognized influencer on continuous testing and DevOps thought leadership, an international speaker, blogger, and also a patent-holding inventor (test exclusion automated mechanisms for mobile J2ME testing). Eran is active in the community and can be found across social media and has his own blog.