How many selenium test scripts can be safely run with Jenkins and Maven

Question

Background

I have a set of Java test scripts that are all in one package. My pom file specifies Testng. I store the tests on Bitbucket along with all the libraries required. I have an instance of Jenkins running on my server as part of my site. The tests run via headless Firefox.

Issue

So it was working perfectly until a larger batch was added. Each .java file has between 5-12 @test declarations (Testng) in it, pretty much each @test is a method. It worked fine when there was about 5 .java files in the Bitbucket but now that there are 14 .java files in there it's throwing an error.

Jenkins View Console output

org.openqa.selenium.firefox.NotConnectedException: Unable to connect to host

firefox output

(process:23458): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' 
failed Xlib:  extension "RANDR" missing on display ":79".
firefox: Fatal IO error 2 (No such file or directory) on X server :79

Deduction

So this looks like an error with the Xvnc that was set up for headless testing since its a GLib-CRITICAL problem and a connection error. But nothing in the Jenkins build has changed from the last time bar the extra tests.

Question

Now for the actual question, Is it save to run large batches of test with Jenkins or should they be broken into sub sets of tests to catch errors like this, in my case would that mean having lots of different Bitbuket repositories so as to run different batches? or can I run lots of batches from one repository?

If anyone has a quick fix or a suggestion on what could fix my error that would be appreciated too, I am just going to roll-back the repository to a runnable state though and add the new tests individually.

Answer 1

It is safer to run your tests through a "Selenium Grid" , even if that grid needs to be on the same machine as the Jenkins server. Better than running multiple browsers directly from the Jenkins job. I think you will get much better results that way. As for the number that you can run, it depends on the power of the system, but running more than 5-10 at a time (threads) is reasonable probably. Also, it simplifies things to have 1 test method per test class but if your willing to deal with the additional work of making sure a browser stays alive through multiple test methods, then you can do it that way, but i would say its prone to error.

Answer 2

Since you're using TestNG, you could try excluding different test cases to try and track down the error. Going by the details you posted, it's hard to tell whether the error is due to a resource constraint or a bad test case.