A few months ago, if you asked me what I believe about adding reporting to an automation project, my answer was “Why do it?”

 

I thought that reports that show the test execution status are useless because Jenkins has plugins that generate them. Why put any effort in creating my own if they are available out-of-the-box?

 

But recently I realized that there are other types of reports that could be very useful.

 

All Selenium tests are created for a QA team so its testers can run them attended or unattended for new builds with bug fixes or new features. Most of the QA people do not know enough coding so they can just read it to find out what the code does. How can they know the steps that a Selenium test goes through without looking in the code?

 

How can they see what the test does, step by step?

 

The answer is by using self-documenting reports for the Selenium test. This is a report that shows all steps that a Selenium test goes through.

 

There is nothing out-of-the-box in Jenkins that does this so I started looking into possible ways of doing it.

 

The first one creates a simple console reporter that writes to the console all steps of a Selenium test and screenshots taken after each step.

The second does the same thing but with the log4j framework so all information is logged in an HTML file.

 The layout of the HTML file is not great which leads to the third solution which is to extend the TestNG HTML reports.

This third attempt is probably good enough but it requires adding code to each page class method to be included in the final report. This is code duplication which we should try to stay away from.

 

Is there a solution that can generate the self-documenting reports with no code changes?

 

There is one called the Allure framework.

This framework does it all for you without adding code to the page methods. You just add annotations to all methods to be added to the final report. The screenshots are added to the same report through a listener.

There are a lot of possible solutions with the last one being the best.

I had a lot of fun (ok, honestly, the Allure framework is well documented but if you ask a lot of it, good luck finding help) working on these different ways of reporting.

So I documented everything in a reporting ebook which you can buy if you want to save time and maybe learn one thing or two.

 

A few words about the 4 reporting solutions:

• all solutions are built with Java, Selenium WebDriver, TestNG and Maven.
• all code is structured using page object model.
• the page classes use a base page class for common page information
• the test class uses a base test class for the driver variable and test fixtures.

 

All solutions implement a simple test case for the Vancouver Public Library site:

1. open the site (http://www.vpl.ca)
2. on the home page, type a keyword in the search box
3. execute the search by clicking the search button
4. on the results page, click the next page to go to page 2
5. on the results page, click the next page to go to page 3
6. on the results page, click the next page to go to page 4
7. on the results page, click the previous page to go to page 3
8. assert that the current page number is 3

 

The goal for each solution is to generate a report such as

• open the site
• type a keyword in the search box
• execute the search
• click the next page
• click the next page
• click the next page
• click the previous page
• assert that the current page number is 3

 

You can see the first solution here. It will give you an idea of the process followed for the other 3.

The last solution is different from the first 3 because it relies not on my own code but on another library. The last solution, the one based on the Allure framework, is the best.

 

The first solution is free for you. The other 3 are not.

If you want all of them (and the full code for each), please make a PAYPAL transfer of USD 9 to alex@alexsiminiuc.com including your email address. An Amazon gift card works as well.