Use Test Factories for Dynamic Tests

Most of the time, we need normal unit tests for test automation:

@Test
public void testSearch() {
}

A normal unit test is a public method that does not return a value, it does not have parameters and uses the @Test annotation.

The unit test can have a parameter though if it is used with a data provider:

@DataProvider(name = "searchKeywords")
public Object[][] searchKeywords() {
return new Object[][] {
{ "iphone"}, { "ipad"}, { "iwatch"}  };
}

@Test(dataProvider = "searchKeywords")
public void testSearch(String keyword) {
}

What is a dynamic test then?

A dynamic test is a unit test of a test class that has a constructor.

The constructor may have a parameter or more and saves them as class members:

import org.testng.annotations.Test;

public class SampleTest {

private int n;

public SampleTest(int n) {
  this.n = n;
}

@Test
public void testScript() {
  System.out.println("test script: " + n);
}

}

The test class gets an integer parameter in the constructor and saves it in a member of the class.

The testScript() method displays the value of the n class member.

testScript() is a dynamic unit test because it belongs to a test class that uses a constructor with a parameter (or more).

How do you use such a unit test?

A test factory creates objects of the test class.

These objects are then executed in the background:

import org.testng.annotations.Factory;

public class FactoryTest {

  @Factory
  public Object[] createTests() {
    System.out.println("create the tests dynamically!");
    Object[] tests = new Object[10];

    for (int i = 0; i < 10; i++)
      tests[i] = new SampleTest(i);

    return tests;

}

}

The createTests method creates first an array to store of test objects of the SampleTest class.

It then creates multiple SampleTest objects and saves them in the array.

Finally, it returns the array of objects.

This is the result of executing the FactoryTest class:

[RemoteTestNG] detected TestNG version 6.14.2
create the tests dynamically!
test script: 0
test script: 1
test script: 5
test script: 3
test script: 9
test script: 2
test script: 7
test script: 6
test script: 8
test script: 4
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript
PASSED: testScript

===============================================
Default test
Tests run: 10, Failures: 0, Skips: 0
===============================================

We see in these results 10 tests being executed and passing.

The interesting part is that they were not executed in the order they were created.

When would dynamic tests be needed in Selenium test automation?

Let’s assume that we want to automate a test case for the https://www.thebay.com/ site.

If you mouse over the accessories option of the menu, multiple categories are displayed.

bay

We want to verify that when clicking on any of these categories, results are displayed.

We could get started with a test as follows:

import static org.testng.Assert.assertTrue;

import java.util.List;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class CategoriesTest {

@DataProvider(name = "accessoriesCategories")
public Object[][] createData1() {
  return new Object[][] {
    { "New Arrivals"},
    { "Best Sellers"},
    { "Online Only"},
    { "Watches"}
  };
}

@Test(dataProvider = "accessoriesCategories")
public void testCategoriesUsingDataProvider(String category) {

  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  ResultsPage resultsPage = menu.selectCategory(category);
  assertTrue(resultsPage.hasResults(), "no results for " + category);

}

}

The categories are defined in a data provider and the test has a parameter for a category.

The test will be executed once for each category.

The test uses 3 classes.

HomePage.java

public class HomePage {

public void open() {
  System.out.println("open home page");
}

public Menu openAccessoriesMenu() {
  System.out.println("open accessories menu");
  return new Menu();
}

}

Menu.java

import java.util.Arrays;
import java.util.List;

public class Menu {

public ResultsPage selectCategory(String category) {
  System.out.println("category selected: " + category);
  return new ResultsPage();
}

public List getCategories() {
  System.out.println("get categories");

  return Arrays.asList( "New Arrivals",
                        "Best Sellers",
                        "Online Only",
                        "Watches");
  }

}

ResultsPage.java

public class ResultsPage {

public boolean hasResults() {
  return true;
}

public HomePage goBack() {
  System.out.println("go back to home page");
  return new HomePage();
}
}

The result of executing the test is below:

[RemoteTestNG] detected TestNG version 6.14.2
open home page
open accessories menu
category selected: New Arrivals
open home page
open accessories menu
category selected: Best Sellers
open home page
open accessories menu
category selected: Online Only
open home page
open accessories menu
category selected: Watches
PASSED: testCategoriesUsingDataProvider("New Arrivals")
PASSED: testCategoriesUsingDataProvider("Best Sellers")
PASSED: testCategoriesUsingDataProvider("Online Only")
PASSED: testCategoriesUsingDataProvider("Watches")

===============================================
Default test
Tests run: 4, Failures: 0, Skips: 0
===============================================

The solution works if the categories for the Accessories option do not change.

But what if they change all the time?

Obviously, we will not be able to use the data provider any longer so the test will change:

@Test
public void testCategoriesWithoutDataProvider() {
  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  List < String > categories = menu.getCategories();

  for (String category: categories) {
    ResultsPage resultsPage = menu.selectCategory(category);
    assertTrue(resultsPage.hasResults(), "no results for " + category);

    homePage = resultsPage.goBack();
    menu = homePage.openAccessoriesMenu();
  }

}

The test is more complicated than before.

It gets all categories of the menu and then iterates through them. For each category, the test selects the category, verifies that there are results and then goes back to HomePage.

The result of executing the test is below:

[RemoteTestNG] detected TestNG version 6.14.2
open home page
open accessories menu
get categories
category selected: New Arrivals
go back to home page
open accessories menu
category selected: Best Sellers
go back to home page
open accessories menu
category selected: Online Only
go back to home page
open accessories menu
category selected: Watches
go back to home page
open accessories menu
PASSED: testCategoriesWithoutDataProvider

===============================================
Default test
Tests run: 1, Failures: 0, Skips: 0
===============================================

The test does pretty much the same thing as before.

But instead of having one test for each category, we have 1 single test for all categories.

If the test fails, it will not be clear which category has a problem.

So this is what we want to fix.

We will simplify the test class so it does only the selection of the category:

import static org.testng.Assert.assertTrue;

import java.util.List;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class SelectCategory {

private String category;

public SelectCategory(String category) {
this.category = category;
}

@Test
public void selectCategory() {
  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  ResultsPage resultsPage = menu.selectCategory(category);
  assertTrue(resultsPage.hasResults(), "no results for " + category);

}
}

And we will use a test factory class that gets all categories and creates multiple objects for the SelectCategory class:

import java.util.List;

import org.testng.annotations.Factory;

public class SelectCategoryFactory {

@Factory
public Object[] testCategories() {
  System.out.println("start test factory");

  HomePage homePage = new HomePage();
  homePage.open();

  Menu menu = homePage.openAccessoriesMenu();

  List < String > categories = menu.getCategories();

  closeBrowser();

  Object[] tests = new Object[categories.size()];

  for (int i = 0; i < categories.size(); i++)
    tests[i] = new SelectCategory(categories.get(i));

  System.out.println("end of test factory");

  return tests;
}

private void closeBrowser() {
  System.out.println("close site");
}

}

The result of executing the test factory shows again 4 tests passing:

[RemoteTestNG] detected TestNG version 6.14.2
start test factory
open home page
open accessories menu
get categories
close site
end of test factory
open home page
open accessories menu
category selected: Watches
open home page
open accessories menu
category selected: Best Sellers
open home page
open accessories menu
category selected: New Arrivals
open home page
open accessories menu
category selected: Online Only
PASSED: selectCategory
PASSED: selectCategory
PASSED: selectCategory
PASSED: selectCategory

===============================================
Default test
Tests run: 4, Failures: 0, Skips: 0
===============================================

The test factory allows the initial test to be changed so that it works for dynamic categories. It also allows executing a test for each category.

Advertisements

Don’t Do It Yourself

DDIY

DIY or Do it yourself.

I started paying attention to the DIY concept after seeing the following method that waits until a page is loaded in the browser:

public void waitForPageToLoad(String string){
  String url = browserDriver.getCurrentUrl();
  Boolean stringExists = url.contains(string);
  int Count = 0;
 
  while (Boolean.FALSE.equals(stringExists) && Count < 100){
     url = browserDriver.getCurrentUrl();
     stringExists = url.contains(string);
     Count = Count + 1;
     try {
        Thread.sleep(500);
     } catch (InterruptedException e) {
         // TODO Auto-generated catch block
         e.printStackTrace();
     }
  }
 
   //if(stringExists==false)
   if(Boolean.FALSE.equals(stringExists))
      throw new IllegalStateException(
        “Page not loaded. Current page” + “is: “
         + browserDriver.getCurrentUrl()); 
   }
}

 

The  method seems useful at first sight. But reading the code with more attention, one realizes that it is just a bad implementation of concepts that already exist in the Selenium WebDriver library: WebDriverWait and ExpectedConditions.

 

The same result is provided by 2 lines of code:

WebDriverWait wait = new WebDriverWait(driver, 30);
wait.until(ExpectedConditions.urlContains(keyword));

 

 

 

When writing code, this is one of the things that are always so tempting.

  1. You need to convert an array into a list so you put together a method that does it.
  2. You need a  way to wait until a page is loaded so you write quickly some code for it.
  3. You need random strings and numbers in your project so you create your class that generates them.

 

After doing it yourself, the result is always the same. You are satisfied about being capable of completing the task. You are capable of writing code that is useful in your project. You can show it to a colleague with pride.

 

But then, you discuss your new classes with someone a bit more senior.

 

Who listens to you carefully and then replies with 3 words:

Don’t do it yourself!

(There are actually 4 words here, hmmm, my counting is not what it used to be.)

 

What did you say? What do you mean by “don’t do it yourself”? What is wrong with my code? It works very well. It has no bugs. It is well designed.

 

That’s what you think now. It may be working for now but will it continue to do so in the future for all your other needs?

Are you sure that it will be easy to extend and maintain?

And actually, why are you doing it? Don’t you have other more important code to write?

 

Yes, I have more code to write. But what about all these things? Generating a random number and string, for example?

 

 

Before starting to write any common-purpose code, check if anyone else did it before you. Chances are very high that there is already a library out there that does what you need and much more. This library was created to work in a variety of situations. It is well designed and probably extensible. It is well documented and well tested.

 

 

Let’s take generating a random string and number.

Your code may look like this:

String text = "";
char[] symbols = new char[] {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'};

int size = 20;

for (int i = 0; i < size; i++) {
Random random = new Random();
int randomIndex = random.nextInt(symbols.length);
text = text + symbols[randomIndex];
}

System.out.println(text);

 

Easy enough, isn’t it?

 

Yes, but very limited too.

 

How about using instead the RandomStringGenerator class from the commons apache library?

Look how easy it is to

  • generate random string values from letters only
  • generate random string values from digits only
  • generate random string values from letters and digits and with variable length
char[] letters = new char[] {'a', 'z'};
char[] numbers = new char[] {'0', '9'};

RandomStringGenerator generator;

generator = new RandomStringGenerator
.Builder().withinRange(letters).build();

System.out.println(generator.generate(20));

generator = new RandomStringGenerator
.Builder().withinRange(numbers).build();

System.out.println(generator.generate(20));

generator = new RandomStringGenerator
.Builder().withinRange(letters, numbers).build();

for (int i = 1; i <=10; i++)
System.out.println(generator.generate(10, 20));

 

In conclusion …

 

There is no need to re-invent the wheel.

Unless you have so many ideas and intuitions that your new wheel is compared to the old one what the first iphone was compared to a blackberry.

 

Otherwise, spend your time to write code that implements a good problem for the company you work for.

And, every time you can, don’t do it yourself.

Use third party libraries instead such as apache commons.

6 things to avoid when learning selenium automation

dontdoit-1

1. Aiming at becoming a Selenium automation expert

This goal is so high that you will probably fail.

You should replace it with something more achievable like “i want to get a test automation job in 1 year”.

 

2. Selenium WebDriver certifications

Selenium WebDriver certifications are scams. Even if they would be good, they wont help you getting hired. When studying for a certification, you study for the exam that you will take and not for the subject.

Passing an exam is a very poor way of measuring your programming skills.

 

I met people who passed with 90% grades TOEFL (test of english as a foreign language) and TWE (test of written english) but speak and write very bad english.

 

3. Jumping into writing Selenium WebDriver tests before learning a programming language

Test automation with Selenium WebDriver is basically programming.

Without knowing a programming language well, you will always have big obstacles in writing automation code. Selenium WebDriver is just a library that can be used through multiple languages.

Jumping to Selenium tests with no programming is similar with trying to cook lasagna with no cooking skills.

Shortcuts never work. There are no shortcuts to math, to learning french, to getting fit, to anything. You have to put the effort in and follow a process that takes time.

 

4. Looking for the best selenium book or online course

There is no such thing. You will find though many great books and online courses.

The best book for Selenium should include everything about Selenium, everything about the programming language (lets say Java), everything about an IDE (example: Eclipse), everything about unit testing, maven, code design, code refactoring, maven, etc.

Do you really think that it exists?

It does not.

To learn test automation with Selenium, you need to use a multitude of resources, books, blogs, youtube videos, online courses, webinars, etc.

Because the amount of information needed for an automation job is very high.

 

5. Learning by yourself

If you attempt learning by yourself, chances are high that you wont make it.

Find other people with the same goal, take instructor-led courses, find developers willing to help you, go to meetups, anything but being by yourself.

 

6. Selenium IDE

Record and play is not test automation

 

What is a selenium tester doing in real time

If you missed the previous day, you can find it here.

Morning of January 18

I start the day with something to bring my mood and energy up.
One cup of hot, strong coffee (no sugar, no milk) for the energy and Rammstein music for the mood.

Today, I need to focus on test stability issues.

I have around 20 Jenkins jobs that run daily automated tests related to various site features.

Some jobs pass consistently but a few fail from time to time.

The first one that I will look at is for the ChangePassword feature.

It runs all tests from the ChangePassword test class using a Maven command:

mvn clean -Dtest=automatedtests.changePassword.ChangePasswordTests test

Out of 5 job runs, there are 2 failures. The Jenkins console log says that assertions are failing. I will run the test class locally to see what is going on.

 

15 minutes later

From all change password tests, the one that changes the password successfully works. All other negative tests that verify that the password cannot be changed (if it is too short or if it includes special characters) fail. Each negative test attempts changing the password and then checks if an error message is displayed.

 

But the error message is all the time the same. It should say that the password is too short or that the password cannot include special characters. Instead, it always says that the passwords are not identical.

 

This seems like an application bug so I should speak to the developers. I will record a video with the tests execution and send it in an email asking for their opinion. Maybe there is a good explanation for it. I better get their opinion first and then, if they agree, log a bug to Jira.

 

10 minutes later

The email is out so I can look into the second failing Jenkins job. This one is about the user acount security questions.

The Jenkins console log points to timeout issues for different page elements.

There are a few possible reasons:

  • The site is very slow.
  • The test does not navigate from a page to the next one. Maybe the test executes the code for clicking a button and the button is not clicked.
  • The ids of the elements changed again.

The html of the site is pretty bad with many elements having either no ids or dynamic ids.

I will have to run these tests locally as well, then confirm which elements cannot be found, get their ids and compare them to the page element ids from Chrome Inspector.

So booooriiiing, does anyone like to do this part?

Anyways, as Maximus’s servant said it in Gladiator  (what a terrific movie! looking forward to watch it again) when asked if he likes what he does:

Maximus: Do you find it hard to do your duty?
Cicero: Sometimes I do what I want to do. The rest of the time, I do what I have to.

I will do as well what I have to.

Reluctantly 😦

 

Later in the day

I am really upset now. All elements from the security questions page have different ids. I need to modify all locators from the SecurityQuestions.java class and retest all tests that use it.

This is such a waste of effort, modifying page element locators over and over and over.

Starting from scratch 🙂

Too bad that the developers are so busy with the new features that no one has time to add the custom ids that I keep asking for.

I am so looking forward to the end of the sprint. Maybe after it, I will be able to pair with a developer and go through all pages and add custom ids to all elements.

Fingers crossed!

 

Finally, the last failing Jenkins job.

This one is for the user registration tests.

The console log shows stale element exceptions for a few elements. This is weird since I am sure that these tests were running very well.

Maybe the developers enabled the auto postback setting for the page elements which makes the element regenerate every time it is interacted with. I solved this before for other pages by retrying a method on exception a few times so I can do the same here. But since it is later in the day, it will have to wait until tomorrow.

 

4 pm already?

Before I go home, someone mentioned on LinkedIn that page factory should not be used any longer.

It is the video of the keynote of a Selenium conference from 2016. I better watch it to get more details.

 

4.30 pm

Yes, Simon Stewart says that no one is using Page Factory and that its design could have been better.

But apparently, no one uses it correctly either and many people try to generate page objects automatically while using page factory.

Wow, never crossed my mind to generate page objects automatically!

So Page Factory works if you use it as it is supposed to be used.

I should watch this video again.

I laughed so much during it, Simon has a great sense of humor 🙂

Use a custom driver class instead of utility classes

How can I create my own driver class in Selenium?
When learning test automation, you will want your test methods to be executed in multiple browsers such as Firefox and Chrome.

A Firefox driver is created with the FirefoxDriver class:

WebDriver driver = new FirefoxDriver();

A Chrome driver is created with the ChromeDriver class:

WebDriver driver = new ChromeDriver();

Both driver objects use the WebDriver type but are instantiated by different classes.

Now, you dont want to change the class from FirefoxDriver to ChromeDriver and back every time you want to run the tests in another browser.

Instead, you would like to create the driver based on the browser name:

WebDriver driver = new BrowserDriver(browserName);

How do we do this?

We create a custom driver class that implements the WebDriver interface and overrides its methods.

public final class BrowserDriver implements WebDriver {  

   private WebDriver driver;
   private final String browserName;
   private final int timeout = 30; 
   private final String chromeDriverPath = "./src/chromedriver.exe";  

   public BrowserDriver(String browserName) { 
      this.browserName = browserName; 
      this.driver = createDriver(browserName); 
   }  

   private WebDriver createDriver(String browserName) { 
      if (browserName.toUpperCase().equals("FIREFOX") 
         return firefoxDriver(); 

      if (browserName.toUpperCase().equals("CHROME") 
         return chromeDriver();  

      throw new RuntimeException ("invalid browser name"); 
   } 

   private WebDriver chromeDriver() { 
      if (!new File(chromeDriverPath).exists()) 
        throw new RuntimeException
                    ("chromedriver.exe does not exist!"); 

      try { 
        System.setProperty("webdriver.chrome.driver", 
                           chromeDriverPath); 
        return new ChromeDriver(); 
      } 

      catch (Exception ex) { 
        throw new RuntimeException
              ("couldnt create chrome driver"); 
      } 
   } 

   private WebDriver firefoxDriver() { 
      try { 
         return new FirefoxDriver(); 
      } 
      catch (Exception ex) {         
         throw new RuntimeException
              ("could not create the firefox driver"); 
     } 
   } 

   @Override 
   public String toString() { 
      return this.browserName; 
   } 

   public WebDriver driver() { 
      return this.driver; 
   }  

   @Override 
   public void close() { 
      driver().close();   
   }  

   @Override 
   public WebElement findElement(By locator) { 
      return driver().findElement(locator); 
   }  

   @Override 
   public List findElements(By arg0) { 
      return driver().findElements(arg0); 
   }  

   @Override 
   public void get(String arg0) { 
      driver().get(arg0); 
   }  

   @Override 
   public String getCurrentUrl() { 
      return driver().getCurrentUrl(); 
   }  

   @Override 

   public String getPageSource() { 
      return driver().getPageSource(); 
   }  

   @Override 
   public String getTitle() { 
      return driver().getTitle(); 
   }  

   @Override 
   public String getWindowHandle() { 
      return driver().getWindowHandle(); 
   }  

   @Override 
   public Set getWindowHandles() { 
     return driver().getWindowHandles(); 
   }  

   @Override 
   public Options manage() { 
      return driver().manage(); 
   }  

   @Override 
   public Navigation navigate() { 
      return driver().navigate(); 
   }  

   @Override 
   public void quit() { 
     driver().quit(); 
   }  

   @Override 
   public TargetLocator switchTo() { 
      return driver().switchTo(); 
   }  

   @Override 
   public  X getScreenshotAs(OutputType target
   throws WebDriverException { 
      return ((TakesScreenshot) driver())
                    .getScreenshotAs(target);
   } 

}

 

The only method that is not defined in the WebDriver interface is createDriver().
createDriver() creates a driver object based on the browser name and stores it in the driver variable.

All other methods just override the WebDriver interface methods.

In the test method, you can now use the new BrowserDriver class instead of FirefoxDriver and ChromeDriver:

WebDriver driver = new BrowserDriver(“CHROME”);

driver.get(“http://www.bestbuy.com”);

WebElement searchBox = driver.findElement(searchBoxLocator);

 

The custom driver class can add as well new methods.

 

Lets say that you want to have find element methods that use the WebDriverWait and ExpectedConditions classes:

public WebElement findVisibleElement(By locator) {
   WebElement element = new WebDriverWait(driver(), timeout)
                .until( visibilityOfElementLocated (locator));
   return element;
}

public WebElement findClickableElement(By locator) {
   WebElement element = new WebDriverWait(driver(), timeout)
               .until( elementToBeClickable (locator));
   return element;
}

public WebElement findHiddenElement(By locator) {
   WebElement element = new WebDriverWait(driver(), timeout)
              .until( presenceOfElementLocated (locator));
   return element;
}

You may be tempted to create a utility class, move these methods there and then use the utility class as a parent for the page classes.

Resist this temptation as it leads to bad things 🙂

Avoid utility classes in your automation projects.

More about utility classes very soon on this blog.

Instead of the utility class, you can move these methods to your custom driver class.

public final class BrowserDriver implements WebDriver {  

   private WebDriver driver; 
   private final String browserName; 
   private final int timeout = 30; 
   private final String chromeDriverPath = "./src/chromedriver.exe"; 

   public BrowserDriver(String browserName) { 
      this.browserName = browserName; 
      this.driver = createDriver(browserName); 
   }  

......................................................................

   @Override 
   public WebElement findElement(By locator) { 
     WebElement element = new WebDriverWait(driver(), timeout) 
                        .until( visibilityOfElementLocated(locator)); 
     return element; 
   }  

   @Override 
   public List findElements(By arg0) { 
     return driver().findElements(arg0); 
   }

   public WebElement findClickableElement(By locator) { 
      WebElement element = new WebDriverWait(driver(), timeout) 
                       .until( elementToBeClickable(locator)); 
      return element; 
   } 

   public WebElement findHiddenElement(By locator) { 
      WebElement element = new WebDriverWait(driver(), timeout) 
                      .until( presenceOfElementLocated(locator)); 
      return element; 
 }

}