Test Automation Framework (Selenium with Java) — Minority Report or Allure in Action

S01E11 of the Test Automation Framework series about everything you’ll need to set up the nice, simple, yet sophisticated framework.

Covered with clear explanations and pretty illustrations.

Sounds like fun? Cool. Now, please, fasten your seatbelts because you’re here for a ride.

S01E01 — What To Automate?

S01E02 — Test Automation Environment and Tools

S01E03 — The First Selenium Test Case

S01E04 — Selenium Foundations Revisited

S01E05 — Page Factory and Elements Related Exceptions

S01E06 — Page Loading Strategies and Waits

S01E07 — Translating JIRA with Selenide (with Exercises)

S01E08 — JIRA, Selenide, Complex SQL, Java Objects with Equals & HashCode (with Exercises)

S01E09 — Code Review and Refactoring (Part 1)

S01E10 — Code Review and Refactoring (Part 2)

In the last episode, we’ve finally finished writing our client-side tests. We covered some complex stuff, like Builder Design Pattern, Rest Assured, or Lombok (learn more: S01E10), we have implemented a simple database connector using the JDBC (learn more: S01E08). It seems like we’re prepared for whatever developers will throw at us.

But, there’s a catch — we have automated Test Cases, but the execution still has to be performed manually, and that’s not how it’s supposed to be.

Before jumping straight into implementing our Test Suite into so-called Continous Integration, we have to prepare some kind of a reporting utility, so that we can monitor the execution of our tests and report the results to the development team and “management people”.

Allure is a no-brainer, at least for me, when it comes to end-to-end test automation reporting. It provides a pretty dashboard, and what’s more important — it’s extremely easy to implement.

Personally, I’m a visual learner, so the more graphs, icons, colors, and interactive elements — the more insight I can extract from the data. On the contrary, I have to admit that my approach is straight up subjective, hence here’s a quote from the Selenide documentation:

QA engineers often want to generate “beautiful” reports. It’s for managers, they say.
I am rather skeptical about those reports. I think managers don’t really read those reports. Think twice before you spend your time on “beautifying” reports.

If you still insist, you can setup Allure. It’s a popular open-source reporting library from Qameta company.
It has built-in integration with Selenide. And yes, its reports look really nice. :)

The choice is yours. If you’d like to apply beautiful reports for your Test Automation Framework go ahead and read the following article.

In case you’d like to follow this guide without the necessity of creating an entire Test Automation Framework feel free to clone the repository from: https://github.com/n4bik/test-automation-framework/tree/Fear-AuthAPI

Secondly, you have to run the Ultimate Stack Developer application on your local machine. Step-by-step tutorial on how to do that can be found in the following article: S01E02 — Test Automation Environment and Tools

Let’s begin with the update of the pom.xml. I’m going to follow Allure’s documentation for the TestNG test framework.

Add the additional dependencies and configuration, as presented in the code snippet below:

<properties>
...
<aspectj.version>1.9.8.RC3</aspectj.version>
<allure.version>2.15.0</allure.version>
...
</properties>
<dependencies>
...
<dependency>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-testng</artifactId>
<version>${allure.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.version}</version>
<configuration>
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
</argLine>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>

Once we’ve updated the pom.xml, let’s check if the Allure is properly implemented. Run the mvn clean test command via Terminal or select the Maven Lifecycle steps from the IntelliJ IDEA’s sidebar (firstly run clean, and then test).

The test step should run the Regression Test Suite defined in the testng.xml file (within the root directory of the Test Automation Framework). Once the Test Run is finished you should see something similar to the picture below.

One more thing is required before we can deploy the Allure reports — we have to install the Allure command-line application locally. To do that simply follow the steps mentioned in Allure’s docs.

Now, we can run the allure serve command from the root directory of the Test Automation Framework.

A short while after, your default Web Browser should open the Allure dashboard and your Test Automation Framework Regression Test Run results.

You can roam freely through the application to familiarize yourself with the Allure.

Looking pretty cool, am I right? Let's see what would happen if there will be a bug within our Test Case.

To force an error I changed the HomePage class’ yellowButton locator:

Below you can see the result of the single Test Case that has failed. There’s one thing, which may be concerning — our Test Steps are not logged within the Allure report. Also, in case of an error, we don’t have screenshots attached to the Test Case which failed, which is not cool, as this is one of the best features that Selenide provides us with out-of-the-box.

Let’s fix them both at once, shall we?

We have to add one additional dependency to our pom.xml file, according to Allure’s docs.

Once, we applied additional dependency we can add the AllureSelenide listener to our BasePage class constructor, as shown below. We’re using the constructor to ensure that this particular listener will be attached to every test.

Let’s rerun our test and check if the fix works.

As you can see both issues (a lack of Test Steps and screenshots) has been resolved. Also, one more pro-tip I wanted to share with you. You can use the Page source to find the root of the bug. Simply extend the Page source with the arrows icon next to it and use the DevTools (press F12 in Chrome to invoke it) to check out the HTML code.

As you can see our class is called “yellow-button”, not “yellow-buttons” and this is why the test fails. That’s a pretty quick way to investigate the locator-based issues, so you may save yourself a lot of time in the future, simply by using the Page source feature of the Allure reports.

We’ve ended up with a nice reporting system, yet it still is quite tough to understand what’s going on if you’re not a developer yourself. In case we’d like to be able to present the results of our Test Runs to the so-called business, we have to translate our descriptions from developer to human.

Example of camel case Test Case titles (left) and Test Steps with default Selenide methods which are quite tech-ish

How can we fix that? Fortunately — it’s really simple and requires only a basic understanding of how to use the Annotations in Java.

According to the Features section of the Allure documentation, we can use multiple different techniques to achieve more non-tech-people-friendly descriptions.

Let’s try it out and fix our HomePageTests test cases with better naming. Let’s start with the Description. Allure provides us with two different options:

  1. The description within the @Test annotation — which changes the display name of the Test Case within the Allure report.
  2. The @Description annotation adds a new Description field to the Allure Test Case details.

Every possible setting is presented in the picture below.

If we use the latter option, we can customize the Test Case the most, and provide as many details as we desire.

Allure report with modified Test Case name (@Test annotation) and additional description (@Description annotation)

The second thing is adding more friendly names of the Test Steps. We can do that using the @Step annotation from within our Page Object Models.

After changes, our HomePage.java should look similar to this:

This is our Allure report after implementing description in @Test annotation, @Description annotation in Test Case, and @Step annotation within the HomePage & CategoriesPage. As you can see, we don’t lose the method that we’re using to perform a particular Test Step with the @Step annotation, so it’s a win-win for both the developer and the management.

There’s no proper way to write Steps descriptions — they should be simple and understandable. Each company/development team has to develop its own standards — that’s one of the reasons why soft skills and teamwork are crucial in IT.

As for “where to put the @Step annotation?”, the rule of thumb is public methods that are being used to perform tests. Also, if the Test Step seems a bit too complicated, it may be an indicator that the method needs a little refactoring to make it more readable.

For instance, let’s take a quick look at the Article Page test regarding getting details for each of the Articles available on the page. If we run the verifyArticlesListWithDataBase() test as is, results will be presented in the Allure as presented in the picture below.

112 sub-steps is a lot. This happens because each Selenide $() or $$() method is treated by the Allure as a separate Test Step. Below you can find the Test Case that is responsible for this little monster.

We can extract the code that is being executed for each row after the line:

To do that we can use the IntelliJ IDEA’s built-in function Extract Method. Simply select code that you’d like to extract, right-click on it and select Refactor > Extract Method… from the context menu.

We can name the new method getArticleDetails(). The only thing left is to add a new @Step annotation for the method, so Allure knows how to handle it. Also, I’ve noticed that we could’ve set the articlesListFromPage as a class-scoped variable so that we don’t need to pass it with each method call for the sake of keeping our code cleaner.

Now, as we’re passing a parameter that comes directly from the forEach() method, we can use the method reference (this::getArticleDetails) instead of the lambda function.

After all these changes, our ArticlePage.java should look like this:

@Log4j2
public class ArticlesPage extends PageWithSubtitle {
private List<Article> articlesListFromPage;

...

@Step("Get articles list from the Articles Page")
public List<Article> getArticleListFromPage() {
log.info("Getting Articles list from Articles page");
articlesListFromPage = new ArrayList<>();
articles
.shouldBe(sizeGreaterThan(0))
.forEach(this::getArticleDetails);
return articlesListFromPage;
}
@Step("Get Article details")
private void getArticleDetails(SelenideElement row) {
List<String> categoryTitlesList = new ArrayList<>();
List<String> categoryTagsList = new ArrayList<>();
row.$$(badgeTagsLocator).forEach(badge -> {
categoryTagsList.add(badge.getText());
categoryTitlesList.add(
badge
.hover()
.find(badgeTitleLocator)
.shouldBe(visible)
.getText());
});
Article articleFromList = Article.builder()
.title(row.find(articleTitleLocator).getText())
.authorFullName(row.find(articleAuthorFullNameLocator).getText())
.publishDate(row.find(articlePublishDateLocator).getText())
.categoryTagList(parseListIntoString(categoryTagsList))
.categoryTitleList(parseListIntoString(categoryTitlesList))
.build();
articlesListFromPage.add(articleFromList);
}

Okay, that’s cool, but how does it look in Allure?

Pretty neat, and if we expand one of the Get Article details sub-steps we can even see what row is being processed. That’s something even better.

Okay, looking good, but it still seems a little bit crowded, as for me at least. In case you’d like to skip the Selenide methods display — you can do it. Simply add includeSelenideSteps(false) method to your BasePage.java.

Report now is more condensed and based only on the @Step annotations within your Test Suite.

Once we know how to make our reports pretty. Let’s try out two different approaches to attach the Test Data to Test Steps.

As you could see in the getArticleDetails(SelenideElement row) when we pass a parameter then its value is available in the Allure.

But what, if we’d like to attach the parameter value to the Test Step description? We can do it with curly brackets ({}), like in the snippet below:

Check the latest Allure report picture above to see the effect (the fourth Test Step).

What about getting the method’s return value, how can we access it from the Allure level? Again, it’s as easy as everything that we have done so far. Just add the @Attachment annotation to the method which value we’d like to see in the report and voilà.

A method with the @Attachment annotation in the Allure report

As an exercise, I want you to add your own annotations for the rest of the Test Suite. If you’re curious how I did that — there’s a link at the bottom of the article with a GitHub repository with the source code.

Now we’re going to cover connecting our Test Cases with the corresponding tickets in the JIRA.

As you might remember from the episode S01E07 — Translating JIRA with Selenide (with Exercises)we based our Test Cases on the JIRA tickets. Allure provides a neat feature that enables connection between these two. Below you can find an overview of the tickets as a reminder of what they looked like.

JIRA Tickets list of the Ultimate Stack Developer project

Let’s dig in. First of all — we need to define the address of our Test Management System (or TMS for short). We can do that by creating new allure.properties file within src/test/resources directory. Fill the file with the code, which you can find below:

This way we can assign either User Stories/Tickets/Tasks and Issues/Bugs.

Now we can add new annotations to our Test Cases @TmsLink (for Tickets) or @Issue (for Issues/Bugs). Here’s an example of how to do that:

That’s it. As you can see — it can’t get any easier. As an exercise, you can do the same for each available Test Case from our Test Automation Framework.

JIRA Board is available here with credentials as follows

  • login: lgafrvynkhicgvjmri@kvhrr.com
  • password: testautomationframework1337

As a result of this annotation, we get links to our Tickets inside of Test Cases in Allure.

Link to the JIRA Ticket (USD-1) inside the Test Case in Allure

Allure and JIRA Love Story — Bug Reporting

I know we’re covering a lot here, but bear with me. In case you haven’t added any Bugs yet, here’s a simple template on how to do it. Feel free to use it in your own projects (developers will be thankful).

Environment:Steps to Reproduce:
1. Step one
2. Step two
...
Expected Result:Actual Result:

Here’s a short guide on how to add a new Bug within JIRA. If you don’t have a Bug issue type available in your own JIRA project — you can add it from Project settings > Issue types > Add issue type

Once, we have Bug available, we can click the Create button on the header and fill the required fields.

Example of the Bug ticket in JIRA (1/2)
Example of the Bug ticket in JIRA (2/2)

Once, we have a ticket, we can assign it to the actual Test Case inside our Test Automation Framework.

Below you can find a new Bug icon with the link to the JIRA Bug ticket. Coolio!

Last but not least — we can also provide Epics, Features, and Stories in the same fashion. Learn more here. We have only one Epic called “USD — Client Side” in our JIRA, so let’s go ahead and try it out.

Epics and related Tasks can be found on the Allure Dashboard and inside the Behaviours tab.

Allure Dashboard featuring Epics
Allure Behaviours tab featuring Epics

As our Test Automation Framework covers only one Epic it’s unnecessary to add such high-level abstraction to the Allure, but I just wanted to show you how you can implement this feature.

Imagine the situation when you run your Test Suite on multiple different browsers and systems. For each Test Run, you’d probably like to get a report, but as for now, we don’t have this type of information anywhere in the Allure report. Let’s take care of that with the Allure Environment file.

One way would be adding additional dependency of the allure-environment-writer, which is basically a simple tool that automates the process of creation of the environment.xml file inside of the allure-results directory.

But I dislike simple solutions and I prefer properties files over the XML. So, here’s how to create your own allure-environment-writer inspired by the AutomatedOwl solution.

In the src/test/java/pl/tomaszbuga/utils package create a new utility class called AllureEnvironmentWriter.java.

package pl.tomaszbuga.utils;import com.codeborne.selenide.WebDriverRunner;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.io.*;
import java.util.Properties;
public class AllureEnvironmentWriter {private static final String ALLURE_RESULTS_PATH = "allure-results/";
private static final String ALLURE_ENV_PROP_PATH = "allure-results/environment.properties";
public static void createEnvironmentPropertiesFile() {
Properties properties = new Properties();
Capabilities capabilities = ((RemoteWebDriver) WebDriverRunner.getWebDriver()).getCapabilities();
createAllureResultsDirectoryIfNotExists();try (OutputStream outputStream = new FileOutputStream(ALLURE_ENV_PROP_PATH)) {
properties.setProperty("Browser", capabilities.getBrowserName());
properties.setProperty("Browser Version", capabilities.getBrowserVersion());
properties.setProperty("OS", System.getProperty("os.name"));
properties.store(outputStream, null);
} catch (IOException e) {
e.printStackTrace();
}
}
private static void createAllureResultsDirectoryIfNotExists() {
new File(ALLURE_RESULTS_PATH).mkdir();
}
}

We’re creating a new Properties object (to store our Key/Value properties). We’d need also the Capabilities object to get the name and version of the WebDriver that we’re using in the Test Run.

Then we’re executing a simple method that creates a new allure-results directory (if it doesn’t exist).

At last, we’re using the try-with-resources block to create a new environment.properties file inside of the allure-results directory with the automatically generated values of the WebDriver and the Operating System.

Once we have our AllureEnvironmentWriter setup, we have to run it before the execution of our Test Suite. To do that — let’s create a new package called utils inside of src/test/java/pl/tomaszbuga/tests. Create a new file within this package and name it BaseTest.java.

We’re going to use TestNG’s @BeforeSuite annotation to run the setUpEnvironment() method once before the entire Test Run.

The main difference between this approach and allure-environment-writer by AutomatedOwl (despite using properties instead of XML) is that — we’re doing stuff automatically based on the WebDriver that we’re using. To do that — we have to actually run the WebDriver before executing the createEnvironmentPropertiesFile() method, so that we can extract the browser name and version out of it. Once we’re done — we’re closing the WebDriver and the Test Suite will run afterward.

At last, we can extend the BaseTest in each Test class, and we’re ready for lift-off.

Now, you have a nice-looking, well-documented Test Results Report with Allure.

What we’ve learned in this episode?

  • Why may we want to implement an additional reporting tool
  • How to implement Allure in Java with TestNG
  • How to implement Allure with Selenide
  • How to customize Allure with Selenide
  • How to use different Allure’s annotations
  • How to connect Allure with Jira Tickets/Bugs
  • How to automate the creation of the Allure Environment file

In case of any questions (I believe there can be one or two of those) — feel free to post them in the comments section below.

All the best,

Tomasz Buga, SDET

www.tomaszbuga.pl

Sources:

GitHub Repository available at: https://github.com/n4bik/test-automation-framework/tree/MinorityReport

All illustrations made by Tomasz Buga

--

--

Software Development Engineer in Tests. Passionate about programming. Experienced, former employee of the insurance industry. Graphic designer by choice.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tomasz Buga

Tomasz Buga

Software Development Engineer in Tests. Passionate about programming. Experienced, former employee of the insurance industry. Graphic designer by choice.