We have set a goal to create a flexible and extendable automated testing framework, which should expand test coverage for as many LeaseWeb applications functionalities as possible. Tests should be also run on CI (Jenkins in our case). We are writing our tests on Java with help of several tools:
- Webdriver is a driver that contains programming interface for controlling all kinds of possible actions in browser.
- TestNG is a testing framework. It structures, groups, and launches tests. It also generates testing reports.
- Maven is a software project management and comprehension tool. It manages all dependencies and different flows for building a project.
1. We created a Maven project with next structure:
src/main/java – contains packages with Page Objects. Each package represents separate application and can contain packages with general methods and objects of the framework.
src/main/resources – here we store config files and other resources that are used in our tests.
src/test/java – contains packages with tests. Each package contains tests related only to one application or a test flow that goes through the couple of applications.
This structure is important for a project compiling in Maven.
2. The main goal was to write tests in a readable manner where each step is one single line of code that clearly describes what it does, and at the same time hides all actions that need to be done by Webdriver “under the hood” in our Page Object classes. By this we make it really easy to read and understand what our test does for every team member – from developers to product managers and even sales people.
3. Test run. We decided to use TestNG for several reasons:
a. People are moving from JUnit to TestNG once they start to write integration and end-to-end tests (e.g. with Selenium). On unit tests level you can put up with JUnit deficiencies, but once you start to write integration and end-to-end tests, JUnit stands in your way and TestNG comes to help you out.
b. No need to create your own Thread objects! You can use @Test annotation with ThreadPoolSize, and invocationCount parameters to run parallel tests.
c. Parameterization. A good testing framework should make it simple, and avoid copy\paste coding. Alas, the default JUnit implementation of the parametrized tests is unusable. In short, JUnit expects all test methods to be parameter-less. Because of this constraint, parameterized tests with JUnit looks awkward because parameters are passed by constructor and kept as fields. People who work with JUnit usually test their code with only one set of parameters. I suspect this is because parameterized tests are unusable with JUnit. And this is bad because such tests are really useful. TestNG allows you to use @DataProvider annotation, which allows you to parameterize your tests however you like and improve your test coverage dramatically with less lines of code in your tests.
d. Parameters to passing the tests:
- Configure your tests so they, for example, invoke GUI methods on different servers/ports?
- Run tests using MySQL database instead of h2?
- Run GUI tests with Opera, Firefox, Chrome (or even IE 6)?
- It is all easy to do with @Parameters annotation in tests.
e. Groups of tests. Each test class (and method) can belong to multiple groups, and with TestNG it’s very easy to run only selected group with @Test (group) annotation. You can easily set in CI which group of tests you want to run – smoke, GUI, API, etc.
f. Test dependencies. It’s not desirable practice with unit tests, but when it comes to integration tests, pretty often you cannot avoid it. This is where TestNG becomes very helpful and easy to use.
g. Reporting. TestNG provides more detailed and helpful reports in comparison with JUnit.
All these settings and tweaks can be specified in textng.xml file, which is used by TestNG to run tests both on your local dev machine and on CI Server. It will declare which tests should be run or skipped and contains information about groups, packages, and classes. This file is located in the root project folder.
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" > <suite name="CAS" verbose="1" > <test name="RegressionLogin" > <classes> <class name="OrderFlowTests"> <methods> <include name="orderFlow" /> </methods> </class> <class name="LoginOppTests" > <methods> <include name="successLoginToOpp" /> </methods> </class> </classes> </test> </suite>
4. Our pom.xml setup.
The beauty of Maven for us, is that it keeps and maintains all dependencies that we use in our project. It makes it much easier to setup the project on a new machine, and also makes a lot easier to run the project on CI. We need at least two plugins to start maven-compiler-plugin и maven-surefire-plugin (in the tags of which we have written our testing.xml). We made few tweaks in the plugin configuration section to show our testing.xml file, and also specify which file naming formats should be considered as tests for TestNG. File is located in to the root project folder.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>NewLeaseWebSite</groupId> <artifactId>NewLeaseWebSite</artifactId> <version>0.0.1-SNAPSHOT</version> <dependencies> <dependency> <groupId>org.testng</groupId> <artifactId>testng</artifactId> <version>6.3.1</version> </dependency> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-firefox-driver</artifactId> <version>2.37.1</version> </dependency> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <version>2.26.0</version> </dependency> </dependencies> <build> <testSourceDirectory>src/tests/java</testSourceDirectory> <resources> <resource> <directory>src/main/resources</directory> <excludes> <exclude>**/*.java</exclude> </excludes> </resource> </resources> <pluginManagement> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.16</version> <configuration> <suiteXmlFiles> <suiteXmlFile>testng.xml</suiteXmlFile> </suiteXmlFiles> <includes> <include>**/Test*.java</include> <include>**/*Tests*.java</include> <include>**/*Tests.java</include> <include>**/*Test.java</include> <include>**/*TestCase.java</include> </includes> </configuration> </plugin> </plugins> </pluginManagement> </build> </project>
As a result, we have a test framework that can be easily extended, should not require much effort for maintenance in the future, and can be run on any CI server with several simple setup steps.