ChatGPT for Test Automation

What is ChatGPT, and how does it work?

What is ChatGPT?

ChatGPT is a state-of-the-art language generation model developed by OpenAI. It is a variant of the GPT (Generative Pretrained Transformer) architecture, trained on a massive corpus of text data to generate human-like text. ChatGPT can perform various language tasks such as answering questions, fixing grammatical mistakes, summarizing documents, and much more. Furthermore, it is designed to continue generating text given an initial prompt, making it well-suited for applications such as chatbots and conversational agents.

ChatGPT is a popular Machine-Learning model trained to chat with people and produce correct-looking results.

What is software test automation?

Software test automation is the process of using software tools to automate software application testing. It involves using software scripts, also known as test cases, to perform a set of tests on the application to detect and fix defects early in the development cycle. The main benefits of test automation include increased efficiency and speed, reduced manual effort and human errors, and improved reliability and consistency of the testing process.

Test automation can be applied to various testing types such as unit, integration, functional, and acceptance. The software tools used for test automation vary based on the testing requirements and can range from open-source to commercial tools.

It is important to note that while test automation can significantly improve the testing process, it is not a replacement for manual testing. Test automation should be used to complement manual testing and not replace it altogether. Additionally, the selection of the right test automation tool and the implementation of effective test automation strategies is crucial for the success of a test automation effort.

As you can see above, it produced an impressively good explanation, with just one minor mix-up.

Does it mean that ChatGPT can be trusted though?

We can see how bad it sometimes can get by giving ChatGPT a simple math statement with even and odd numbers, the concept that is typically introduced in second-grade school.

The odd numbers in this group add up to an even number: 15, 32, 5, 13, 82, 7, 1. Is this correct?

Yes, that is correct. The odd numbers in the group are 15, 5, 13, 7, and 1. If we add these numbers together, we get: 15 + 5 + 13 + 7 + 1 = 41. So the sum of the odd numbers is 41. However, 32 and 82 are even numbers in the group. If we add them to the sum of the odd numbers, we get: 41 + 32 + 82 = 155. Since the sum is even, the statement "The odd numbers in this group add up to an even number" is true.

Here, we see that ChatGPT gave us the wrong answer, although the response might look legit at a first glance.

So how does ChatGPT work? It is a neural network trained on a massive amount of data to answer questions and hold context to a certain degree. In essence, it is a set of functions extrapolating based on "training points".

This is where ChatGPT's pitfalls and limitations arise: since it is not nearly as powerful as a human brain, it does not have a high enough degree of complexity to be able to validate and understand either the question or the answer, and therefore often produces non-sensical results that look legitimate. It becomes especially dangerous when most part of the answer looks correct, but the issues are somewhere in the middle of the text. In fact, Stack Overflow has recently banned ChatGPT for "constantly giving wrong answers".

What is testRigor?

testRigor is a human-emulator system that gets a subset of plain English as input and executes it exactly the same way a human would. So, the instructions could look like this:

login
create a new lead
select "Mr" from "Salutation"
generate unique name, then enter into "First Name" and save as "firstName"
enter "Einstein" into "Last Name"
enter "1234567890" into "Phone"
enter "testRigor" into "Company"
click "Save"
check that page contains "created"
check that page contains stored value "firstName"

How does ChatGPT work for code?

Similarly to the description above, ChatGTP can produce some results based on its training that can be very helpful in jump-starting your test.

Here is an example of a sign-up test in Selenium using Java:

Write sign up test in Selenium Java

Here's an example of a sign-up test in Selenium Java:

java

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

public class SignUpTest {
  private WebDriver driver;

  @BeforeMethod
  public void setUp() {
    System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
    driver = new ChromeDriver();
  }

  @Test
  public void testSignUp() {

    driver.get("https://www.example.com/signup");

    WebElement emailField = driver.findElement(By.id("email"));
    emailField.sendKeys("testuser@example.com");

    WebElement passwordField = driver.findElement(By.id("password"));
    passwordField.sendKeys("password");

    WebElement signUpButton = driver.findElement(By.id("signup"));
    signUpButton.click();

    WebElement successMessage = driver.findElement(By.id("success-message"));
    assert.assertEquals(successMessage.getText(), "Sign up successful!");
  }

  @AfterMethod
  public void tearDown() {
    driver.quit();
  }
}

This code uses the ChromeDriver to automate the browser, and TestNG to define the test. The test performs the following steps:

• Navigates to the sign-up page.
• Enters the email, and password.
• Clicks the sign-up button.
• Verifies that the sign-up was successful by checking for the presence of a "Welcome" message.

In this example, we use the ChromeDriver class from the Selenium library to launch the Chrome browser and interact with the website. The @BeforeMethod and @AfterMethod annotations are used to set up and tear down the WebDriver instance before and after each test.

In the testSignUp method, we navigate to the sign-up page, locate the email, password, and sign-up button elements on the page using the findElement method, enter values into the email and password fields, and click the sign-up button.

Finally, we locate the success message element and use the assertEquals method from the TestNG library to verify that the text of the success message is "Sign up successful!".

• It relies on ids being in the code.
• It doesn't set the default timeout.
• It misses clicking on “Sign up” button.
• It doesn't generate a unique username/email so it will fail a second time in a typical test environment.
• It is incomplete. It doesn't validate the email confirmation, and it doesn't validate that the newly created credentials can be used.

Here is how a similar test would look in testRigor:

click "Sign up"
generate unique name, then enter into "Username" and save as "username"
generate unique email, then enter into "Email" and save as "generatedEmail"
generate unique name, then enter into "Name" and save as "generatedName"
enter "PasswordSuperSecure!" into "Password"
save value "PasswordSuperSecure!" as "password"
click "Submit"
check that email to stored value "generatedEmail" was delivered
click "Confirm email"
check that page contains "Email was confirmed"
check that page contains expression "Hello, ${generatedName}"
click "Logout"
login
check that page contains stored value "generatedName"

As you can see above, in testRigor, you can outline the steps and follow them the way you described. In some sense, testRigor is ChatGPT for testing, but with a different purpose: its goal is to allow you create high quality test cases, precisely execute test steps without writing any code or relying on implementation details.

Intrigued? Would you like to learn more?