Category / TestNG

by Alex Siminiuc

Use Test Factories for Dynamic Tests

Most of the time, we need normal unit tests for test automation:

@Test
public void testSearch() {
}

A normal unit test is a public method that does not return a value, it does not have parameters and uses the @Test annotation.

The unit test can have a parameter though if it is used with a data provider:

@DataProvider(name = "searchKeywords")
public Object[][] searchKeywords() {
return new Object[][] {
{ "iphone"}, { "ipad"}, { "iwatch"}  };
}

@Test(dataProvider = "searchKeywords")
public void testSearch(String keyword) {
}

What is a dynamic test then?

A dynamic test is a unit test of a test class that has a constructor.

The constructor may have a parameter or more and saves them as class members:

import org.testng.annotations.Test;

public class SampleTest {

private int n;

public SampleTest(int n) {
  this.n = n;
}

@Test
public void testScript() {
  System.out.println("test script: " + n);
}

}

The test class gets an integer parameter in the constructor and saves it in a member of the class.

The testScript() method displays the value of the n class member.

testScript() is a dynamic unit test because it belongs to a test class that uses a constructor with a parameter (or more).

How do you use such a unit test?

A test factory creates objects of the test class.

These objects are then executed in the background:

import org.testng.annotations.Factory;

public class FactoryTest {

  @Factory
  public Object[] createTests() {
    System.out.println("create the tests dynamically!");
    Object[] tests = new Object[10];

    for (int i = 0; i < 10; i++)
      tests[i] = new SampleTest(i);

    return tests;

}

}

The createTests method creates first an array to store of test objects of the SampleTest class.

It then creates multiple SampleTest objects and saves them in the array.

Finally, it returns the array of objects.

This is the result of executing the FactoryTest class:

[RemoteTestNG] detected TestNG version 6.14.2
create the tests dynamically!
test script: 0
test script: 1
test script: 5
test script: 3
test script: 9
test script: 2
test script: 7
test script: 6
test script: 8
test script: 4
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript

===============================================
Default test
Tests run: 10, Failures: 0, Skips: 0
===============================================

We see in these results 10 tests being executed and passing.

The interesting part is that they were not executed in the order they were created.

When would dynamic tests be needed in Selenium test automation?

Let’s assume that we want to automate a test case for the https://www.thebay.com/ site.

If you mouse over the accessories option of the menu, multiple categories are displayed.

bay

We want to verify that when clicking on any of these categories, results are displayed.

We could get started with a test as follows:

import static org.testng.Assert.assertTrue;

import java.util.List;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class CategoriesTest {

@DataProvider(name = "accessoriesCategories")
public Object[][] createData1() {
  return new Object[][] {
    { "New Arrivals"},
    { "Best Sellers"},
    { "Online Only"},
    { "Watches"}
  };
}

@Test(dataProvider = "accessoriesCategories")
public void testCategoriesUsingDataProvider(String category) {

  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  ResultsPage resultsPage = menu.selectCategory(category);
  assertTrue(resultsPage.hasResults(), "no results for " + category);

}

}

The categories are defined in a data provider and the test has a parameter for a category.

The test will be executed once for each category.

The test uses 3 classes.

HomePage.java

public class HomePage {

public void open() {
  System.out.println("open home page");
}

public Menu openAccessoriesMenu() {
  System.out.println("open accessories menu");
  return new Menu();
}

}

Menu.java

import java.util.Arrays;
import java.util.List;

public class Menu {

public ResultsPage selectCategory(String category) {
  System.out.println("category selected: " + category);
  return new ResultsPage();
}

public List getCategories() {
  System.out.println("get categories");

  return Arrays.asList( "New Arrivals",
                        "Best Sellers",
                        "Online Only",
                        "Watches");
  }

}

ResultsPage.java

public class ResultsPage {

public boolean hasResults() {
  return true;
}

public HomePage goBack() {
  System.out.println("go back to home page");
  return new HomePage();
}
}

The result of executing the test is below:

[RemoteTestNG] detected TestNG version 6.14.2
open home page
open accessories menu
category selected: New Arrivals
open home page
open accessories menu
category selected: Best Sellers
open home page
open accessories menu
category selected: Online Only
open home page
open accessories menu
category selected: Watches
PASSED: testCategoriesUsingDataProvider("New Arrivals")
PASSED: testCategoriesUsingDataProvider("Best Sellers")
PASSED: testCategoriesUsingDataProvider("Online Only")
PASSED: testCategoriesUsingDataProvider("Watches")

===============================================
Default test
Tests run: 4, Failures: 0, Skips: 0
===============================================

The solution works if the categories for the Accessories option do not change.

But what if they change all the time?

Obviously, we will not be able to use the data provider any longer so the test will change:

@Test
public void testCategoriesWithoutDataProvider() {
  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  List < String > categories = menu.getCategories();

  for (String category: categories) {
    ResultsPage resultsPage = menu.selectCategory(category);
    assertTrue(resultsPage.hasResults(), "no results for " + category);

    homePage = resultsPage.goBack();
    menu = homePage.openAccessoriesMenu();
  }

}

The test is more complicated than before.

It gets all categories of the menu and then iterates through them. For each category, the test selects the category, verifies that there are results and then goes back to HomePage.

The result of executing the test is below:

[RemoteTestNG] detected TestNG version 6.14.2
open home page
open accessories menu
get categories
category selected: New Arrivals
go back to home page
open accessories menu
category selected: Best Sellers
go back to home page
open accessories menu
category selected: Online Only
go back to home page
open accessories menu
category selected: Watches
go back to home page
open accessories menu
PASSED: testCategoriesWithoutDataProvider

===============================================
Default test
Tests run: 1, Failures: 0, Skips: 0
===============================================

The test does pretty much the same thing as before.

But instead of having one test for each category, we have 1 single test for all categories.

If the test fails, it will not be clear which category has a problem.

So this is what we want to fix.

We will simplify the test class so it does only the selection of the category:

import static org.testng.Assert.assertTrue;

import java.util.List;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class SelectCategory {

private String category;

public SelectCategory(String category) {
this.category = category;
}

@Test
public void selectCategory() {
  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  ResultsPage resultsPage = menu.selectCategory(category);
  assertTrue(resultsPage.hasResults(), "no results for " + category);

}
}

And we will use a test factory class that gets all categories and creates multiple objects for the SelectCategory class:

import java.util.List;

import org.testng.annotations.Factory;

public class SelectCategoryFactory {

@Factory
public Object[] testCategories() {
  System.out.println("start test factory");

  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  List < String > categories = menu.getCategories();

  closeBrowser();

  Object[] tests = new Object[categories.size()];

  for (int i = 0; i < categories.size(); i++)
    tests[i] = new SelectCategory(categories.get(i));

  System.out.println("end of test factory");

  return tests;
}

private void closeBrowser() {
  System.out.println("close site");
}

}

The result of executing the test factory shows again 4 tests passing:

[RemoteTestNG] detected TestNG version 6.14.2
start test factory
open home page
open accessories menu
get categories
close site
end of test factory
open home page
open accessories menu
category selected: Watches
open home page
open accessories menu
category selected: Best Sellers
open home page
open accessories menu
category selected: New Arrivals
open home page
open accessories menu
category selected: Online Only
PASSED: selectCategory
PASSED: selectCategory
PASSED: selectCategory
PASSED: selectCategory

===============================================
Default test
Tests run: 4, Failures: 0, Skips: 0
===============================================

The test factory allows the initial test to be changed so that it works for dynamic categories. It also allows executing a test for each category.

by Alex Siminiuc

How to retry automatically Selenium tests

Test automation methods fail often.

Sometimes because the test automation code has bugs so you need to fix them.

Other times because the tested site does not load correctly or loads very slowly.

 

Lets look in more detail at this second case.

 

The test automation code is correct.

You prove it correct by running the test multiple times.
If the test method fails a lot (4-5 times out of 10), the problem is in the automation code.
If the test method fails once every 15 or 20 runs, the site is at fault.

 

These rare and random failures are not real failures.

It would be good if we could re-run failed test methods automatically a few times before declaring them as failed.

How can we do this?

Lets start with a simple test class with 3 test methods:

public class TestRetry {

  @Test
  public void testPassed() {
    System.out.println("test passed!");
    assertTrue(true);
  }

  @Test
  public void testFailed() {
    System.out.println("test failed!");
    assertTrue(false);
  }

  @Test
  public void testMayFail() {
    int i = new Random().nextInt(10);
    System.out.println("test may fail! - " + i);
    if (i > 5)
       assertTrue(false);
    else
       assertTrue(true);
  }
}

 

The first method passes all the time.

The second method fails all the time.

The third method fails sometimes.

 

We want to be able to re-run the third test method automatically every time it fails.

 

For this purpose, 2 new classes are created:

  1. RetryAnalyzer (which implements the IRetryAnalyzer TestNG interface)
  2. RetryTestListener (which extends the TestListenerAdapter TestNG class)

 

RetryAnalyzer uses a variable for the current retry count.

It has also a variable for the maximum retry count.

Its retry() method checks if the current retry count is smaller than the max count.

If it is, the current count is increased by 1 and the method returns true.

Otherwise, the method returns false.

 

import org.testng.IRetryAnalyzer;

import org.testng.ITestResult;


public class RetryAnalyzer implements IRetryAnalyzer  { 

  private int count = 0; 

  private int maxCount = 2;

  @Override

  public boolean retry(ITestResult result) { 

    if(count < maxCount) {  

       count++;

       return true;        

    }        

    return false; 

 }

}

 

The RetryAnalyzer class does not do anything by itself.

It needs to be used with the RetryTestListener class.

 

RetryTestListener overrides the onTestFailure() method so that

  1. when the script fails, the script is retried
  2. if the retry count is less than the max count, the script’s status is changed to skipped
  3. if the retry count is equal to max count, the script fails and the script’s status is failed

 

import org.testng.ITestResult;

import org.testng.Reporter;

import org.testng.TestListenerAdapter;

public class RetryTestListener extends TestListenerAdapter  {

@Override
public void onTestFailure(ITestResult result {

     Reporter.setCurrentTestResult(result);

     if(result.getMethod().getRetryAnalyzer().retry(result))                         result.setStatus(ITestResult.SKIP);

     Reporter.setCurrentTestResult(null);

  }

}

 

The test class can be updated now to use the new classes.

The listener is attached to the test class using the @Listeners annotation.

The third test method is updated by adding the retryAnalyzer attribute to the @Test annotation.

 

import static org.testng.Assert.assertTrue;

import java.util.Random;

import org.testng.annotations.Listeners;

import org.testng.annotations.Test;

@Listeners(RetryTestListener.class)

public class TestRetry {

  @Test

  public void testPassed() {

      System.out.println("test passed!");

      assertTrue(true);

  }

  @Test

  public void testFailed() {

      System.out.println("test failed!");

      assertTrue(false);

  }

  @Test(retryAnalyzer=RetryAnalyzer.class)

  public void testMayFail() {

       int i = new Random().nextInt(10);

       System.out.println("test may fail! - " + i);

       if (i > 5)

             assertTrue(false);

      else

             assertTrue(true);

   }

}

 

If the third method fails, its status is set to SKIPPED and the method is retried.

 

For the first retry, if the method passes, the method’s execution ends with the status PASSED.

If the method fails again, the method’s status is set to SKIPPED and the method is retried (retry count = 1).

 

For the second retry, if the method passes, the method’s ends with the status PASSED.

If the method fails, the method’s status is changed to SKIPPED and the method is retried (retry count = 2).

 

Finally, If the method passes, the method’s ends with the status PASSED.

If the method fails, the method’s status is changed to FAILED.