Test automation methods fail often.

Sometimes because the test automation code has bugs so you need to fix them.

Other times because the tested site does not load correctly or loads very slowly.

 

Lets look in more detail at this second case.

 

The test automation code is correct.

You prove it correct by running the test multiple times.
If the test method fails a lot (4-5 times out of 10), the problem is in the automation code.
If the test method fails once every 15 or 20 runs, the site is at fault.

 

These rare and random failures are not real failures.

It would be good if we could re-run failed test methods automatically a few times before declaring them as failed.

How can we do this?

Lets start with a simple test class with 3 test methods:

public class TestRetry {

  @Test
  public void testPassed() {
    System.out.println("test passed!");
    assertTrue(true);
  }

  @Test
  public void testFailed() {
    System.out.println("test failed!");
    assertTrue(false);
  }

  @Test
  public void testMayFail() {
    int i = new Random().nextInt(10);
    System.out.println("test may fail! - " + i);
    if (i > 5)
       assertTrue(false);
    else
       assertTrue(true);
  }
}

 

The first method passes all the time.

The second method fails all the time.

The third method fails sometimes.

 

We want to be able to re-run the third test method automatically every time it fails.

 

For this purpose, 2 new classes are created:

1. RetryAnalyzer (which implements the IRetryAnalyzer TestNG interface)
2. RetryTestListener (which extends the TestListenerAdapter TestNG class)

 

RetryAnalyzer uses a variable for the current retry count.

It has also a variable for the maximum retry count.

Its retry() method checks if the current retry count is smaller than the max count.

If it is, the current count is increased by 1 and the method returns true.

Otherwise, the method returns false.

 

import org.testng.IRetryAnalyzer;

import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer  { 

  private int count = 0; 

  private int maxCount = 2;

  @Override

  public boolean retry(ITestResult result) { 

    if(count < maxCount) {  

       count++;

       return true;        

    }        

    return false; 

 }

}

 

The RetryAnalyzer class does not do anything by itself.

It needs to be used with the RetryTestListener class.

 

RetryTestListener overrides the onTestFailure() method so that

1. when the script fails, the script is retried
2. if the retry count is less than the max count, the script’s status is changed to skipped
3. if the retry count is equal to max count, the script fails and the script’s status is failed

 

import org.testng.ITestResult;

import org.testng.Reporter;

import org.testng.TestListenerAdapter;

public class RetryTestListener extends TestListenerAdapter  {

@Override
public void onTestFailure(ITestResult result {

     Reporter.setCurrentTestResult(result);

     if(result.getMethod().getRetryAnalyzer().retry(result))                         result.setStatus(ITestResult.SKIP);

     Reporter.setCurrentTestResult(null);

  }

}

 

The test class can be updated now to use the new classes.

The listener is attached to the test class using the @Listeners annotation.

The third test method is updated by adding the retryAnalyzer attribute to the @Test annotation.

 

import static org.testng.Assert.assertTrue;

import java.util.Random;

import org.testng.annotations.Listeners;

import org.testng.annotations.Test;

@Listeners(RetryTestListener.class)

public class TestRetry {

  @Test

  public void testPassed() {

      System.out.println("test passed!");

      assertTrue(true);

  }

  @Test

  public void testFailed() {

      System.out.println("test failed!");

      assertTrue(false);

  }

  @Test(retryAnalyzer=RetryAnalyzer.class)

  public void testMayFail() {

       int i = new Random().nextInt(10);

       System.out.println("test may fail! - " + i);

       if (i > 5)

             assertTrue(false);

      else

             assertTrue(true);

   }

}

 

If the third method fails, its status is set to SKIPPED and the method is retried.

 

For the first retry, if the method passes, the method’s execution ends with the status PASSED.

If the method fails again, the method’s status is set to SKIPPED and the method is retried (retry count = 1).

 

For the second retry, if the method passes, the method’s ends with the status PASSED.

If the method fails, the method’s status is changed to SKIPPED and the method is retried (retry count = 2).

 

Finally, If the method passes, the method’s ends with the status PASSED.

If the method fails, the method’s status is changed to FAILED.