Parallel Automated Testing Using Selenium Grid With Jenkins
Everyone who is involved in automated testing and particularly in writing the tests sooner or later faces the same well-known problem - TIME. The time spent on passing all the tests. It grows in direct proportion to the number of writen scenarios and in no way is going to decrease. The number of tests is growing - it’s good. We cover more and more features - it's also nice. Time is growing - it is bad.
It often happens, that the software developers stop to run the tests after changing the code, complaining about the lack of time (and they can be understood). As a result we may find ourselves in a situation when we have tests and at the same time there are lots of bugs that were supposed to be identified by these tests. This is something that our team was concerned with and we have worked out a functional solution of this issue. It was decided to run tests in several threads and remotely, so that they didn't interfere the work. This was achieved via introducing the best parallel testing techniques by means of Jenkins and Selenium Grid.
The following article provides a based on our experience guide on using Selenium grid with Jenkins for distributing tests on several machines with a special focus on the parallel testing for Ruby on Rails apps.
So, let's start. We have:
- Ruby on Rails project
- Cucumber with Capybara tests
- Jenkins, customized on passing browser tests in a single thread
- Many computers on the same local network
Selenium Grid in brief
Selenium Grid is a powerful tool that allows to extend the load of your tests on multiple machines. It provides the ability to manage a number of environments from a central point, which makes it easy to run the tests with a wide combination of web browsers and operating systems. It is open source software and can be used at no charge.
Executable file of Selenium Grid is selenium_server_standalone.jar which contains all needed distributions, namely selenium hub (center of monitoring and management) and selenium node (remote browsers).
Starting Selenium hub
To start it we need to download file selenium-server.jar to our server and run the hub with the following command:
java -jar selenium-server.jar -role hub
The command is executed from a directory that contains the jar file, or run in the team added the path to the file. By default the hub runs on port 4444. If needed, you can specify the port manually by adding the optional key -port to the start-up command:
java -jar selenium-server.jar -role hub -port 4445
The server also supports other ways to start, which can be found in the manual.
After starting the hub you can view its GUI console, by navigating to http://127.0.0.1:4445/grid/console in a browser.
It looks like this:
Starting Selenium node
Now we need to start the nodes. In other words, nodes are the browsers which will run our tests. Enter the following command to start the nodes:
java -jar selenium-server.jar -role node -hub http://localhost:4445/grid/register
By default the nodes run on port 5555. It also can be changed by specifying the optional parameter -port. Keep in mind that the start command contains the key of our running hub. Thus, if we changed the port at the hub start-up it should be changed when starting a node too:
java -jar selenium-server.jar -role node -hub http://localhost:4445/grid/register -port 5556
The start command also has a key -role. If you put it as the node it will be launched for WebDriver (Selenium2) and Remote Control (Selenium). If you are not using Remote Control it makes sense to change the setting of this key to webdriver:
java -jar selenium-server.jar -role webdriver -hub http://localhost:4445/grid/register -port 5556
If you go to the Grid Console http://127.0.0.1:4445/grid/console you should see the following:
It means that we can run our tests in five threads for Firefox, five threads for Chrome and one thread for IE.
If you do not need such set of browsers, it can be configured by the optional key
-browser browserName=firefox, maxInstances=5
Since our server does not have ‘X’, they should be run simultaneously with the launch of the node or we will not be able to connect to our browsers. Our final command to run the node is:
export DISPLAY=:0.0 && xvfb-run –auto-servernum –server-num=0 java -jar selenium-server.jar -role webdriver -hub http://localhost:4445/grid/register -port 5556
So what do we have?
Launched hub and node running on a single instance. But we have a local network with many working machines, and Selenium Grid provides the ability to distribute the performance of tests on other machines.
We can connect via ssh to another machine on the network, install Firefox and Chrome, copy our selenium-server.jar and run it in node mode on the machine. In our case, in addition to the server, we have launched nodes on two more remote machines. Now our Grid Console looks like this:
On the remote machines we intentionally restricted nodes - Chrome 2 pcs, Firefox 2 pcs and excluded IE, as we are not going to use it. So, we can run our tests in 9 threads for Firefox and 9 threads for Chrome.
Creating upstart services to run Selenium grid
We can use an upstart to run Selenium hub and Selenium node. First of all it is right, then it is quite conveniant. To create a Selenium hub service we perform:
Enter the following settings in file:
1. # Selenium Server Standalone 2. 3. description "run Selenium Server as hub" 4. 5. start on runlevel  6. stop on runlevel [!2345] 7. 8. respawn 9. 10. exec java -jar /usr/lib/selenium-server.jar -role hub -port 4445
Now we can use the following commands for the Selenium hub.
To start service:
sudo service selenium-hub start
To stop service:
sudo service selenium-hub stop
To restart service:
sudo service selenium-hub restart
A similar config file can be created for the Selenium node.
1. # Selenium Server Standalone 2. 3. description "run Selenium Server as node" 4. 5. start on runlevel  6. stop on runlevel [!2345] 7. 8. respawn 9. 10. exec export DISPLAY=:0.0 && java -jar selenium-server.jar -role webdriver -hub http://localhost:4445/grid/register -port 5556
Setting up a test environment env.rb
Then we need to specify that our tests should go to remote browsers. To do this, we should add some settings to env.rb file:
Now, when you run the tests, you should use a command with new variable, or even two variables SELENIUM and BROWSER. Key SELENIUM is mandatory. It allows to say that we use a different environment. BROWSER parameter is optional. By default, our tests will be carried out under the control of Firefox. If we want to run them under the control of Chrome we should specify the environment variable BROWSER = chrome.
For the relative Jenkins integration with Selenium, we go to Jenkins and change our command for starting tests: SELENIUM=remote bundle exec cucumber to run tests under Firefox, SELENIUM=remote BROWSER=chrome bundle exec cucumber to run tests under Chrome.
At this stage, our tests will run on a remote browser, but only in one thread.
Installing parallel_test gem
To make our server perform the tests in multiple threads, we need to split tests into groups and exactute each group in its own dedicated process and database. For this purpose, we can use a parallel_tests gem. The installation is standard. We add gem
"parallel_tests", :group => :development in Gemfile. Then go to the build settings in Jenkins. We need to create additional databases to run the tests.
1. bundle exec rake db:create # create a test db 2. bundle exec rake db:migrate # run migrations to the test db 3. bundle exec rake parallel:create # create 8 additional db 4. bundle exec rake parallel:prepare # prepare additional db
and we change the launch command in accordance to the gem instruction:
SELENIUM=remote BROWSER=chrome bundle exec rake parallel:features
It also makes sense to stop an Xvfb before starting the build, as we run it together with nodes.
Now our build will be carried out in 8 threads under the control of Chrome. When passing the test, you can open Grid Console and observe the process of parallelization.
This is more or less all about running Selenium tests for Ruby on Rails apps. We hope our readers will share our excitement about parallel automated testing.
In our case, we have managed to reduce the time of the tests from 50 minutes in a single thread on the local machine to 8.8 minutes in 10 threads in the remote browser. And 10 threads are not the limit, if you have a large network.
While running nodes on the remote machines, we were watching CPU loading on perfoming tests - it was definitely a pleasure. Two processes can run on the same machine with no problems at all.
Special thanks to Dmitry Lyalyuevu for co-authoring.