GitLab Runner Inconsistency: exec-maven-plugin:1.6.0:java org.apache.http.conn.HttpHostConnectException

Question

We have an automation project on GitLab that uses our dedicated "shell" GitLab runner. Project itself is a Selenium project that runs Cucumbers tests and generates a report. It was recently running on Jenkins with no problem whatsoever. To simulate the Jenkins environment, I created a shell GitLab runner installed Java, Maven etc. on the machine with same versions. Dug up plenty of resources online. Modified everything all the way from my "hosts" file. But not managed to solve this issue.

Tests are running fine and generate the report. But the "generate report part" is inconsistent. Meaning that sometimes it's not able to generate the Cucumber report. Two days it's working fine and the next day it bumps to a wall.

Section of the pom file:

<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>

The problem is not the execution of the tests, they're always successful. Problem occurs afterwards. Cucumber cannot generate the report every once in few builds even though nothing has changed neither code-wise nor infrastructure-wise.

This is the error we get after the tests are succesful:

[WARNING] 
org.openqa.selenium.WebDriverException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:23723 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'ip-XcensoredX', ip: 'ip-XcensoredX', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-1051-aws', java.version: '11.0.4'
Driver info: driver.version: RemoteWebDriver
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute (DriverCommandExecutor.java:91)
    at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:637)
    at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:694)
    at org.openqa.selenium.remote.RemoteWebDriver.quit (RemoteWebDriver.java:533)
    at com.company.testautomation.util.WebDriverUtils.shutDown (WebDriverUtils.java:589)
    at com.company.testautomation.CucumberRunner.<init> (CucumberRunner.java:142)
    at com.company.testautomation.CucumberRunner.main (CucumberRunner.java:69)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:834)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:23723 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect (DefaultHttpClientConnectionOperator.java:159)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect (PoolingHttpClientConnectionManager.java:359)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute (MainClientExec.java:381)
    at org.apache.http.impl.execchain.MainClientExec.execute (MainClientExec.java:237)
    at org.apache.http.impl.execchain.ProtocolExec.execute (ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute (RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute (RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute (InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:72)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:56)
    at org.openqa.selenium.remote.internal.ApacheHttpClient.fallBackExecute (ApacheHttpClient.java:139)
    at org.openqa.selenium.remote.internal.ApacheHttpClient.execute (ApacheHttpClient.java:87)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute (HttpCommandExecutor.java:161)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute (DriverCommandExecutor.java:82)
    at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:637)
    at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:694)
    at org.openqa.selenium.remote.RemoteWebDriver.quit (RemoteWebDriver.java:533)
    at com.company.testautomation.util.WebDriverUtils.shutDown (WebDriverUtils.java:589)
    at com.company.testautomation.CucumberRunner.<init> (CucumberRunner.java:142)
    at com.company.testautomation.CucumberRunner.main (CucumberRunner.java:69)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:834)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect (Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect (AbstractPlainSocketImpl.java:399)
    at java.net.AbstractPlainSocketImpl.connectToAddress (AbstractPlainSocketImpl.java:242)
    at java.net.AbstractPlainSocketImpl.connect (AbstractPlainSocketImpl.java:224)
    at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:403)
    at java.net.Socket.connect (Socket.java:591)
    at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket (PlainConnectionSocketFactory.java:75)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect (DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect (PoolingHttpClientConnectionManager.java:359)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute (MainClientExec.java:381)
    at org.apache.http.impl.execchain.MainClientExec.execute (MainClientExec.java:237)
    at org.apache.http.impl.execchain.ProtocolExec.execute (ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute (RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute (RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute (InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:72)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:56)
    at org.openqa.selenium.remote.internal.ApacheHttpClient.fallBackExecute (ApacheHttpClient.java:139)
    at org.openqa.selenium.remote.internal.ApacheHttpClient.execute (ApacheHttpClient.java:87)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute (HttpCommandExecutor.java:161)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute (DriverCommandExecutor.java:82)
    at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:637)
    at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:694)
    at org.openqa.selenium.remote.RemoteWebDriver.quit (RemoteWebDriver.java:533)
    at com.company.testautomation.util.WebDriverUtils.shutDown (WebDriverUtils.java:589)
    at com.company.testautomation.CucumberRunner.<init> (CucumberRunner.java:142)
    at com.company.testautomation.CucumberRunner.main (CucumberRunner.java:69)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:834)
[WARNING] thread Thread[OkHttp ConnectionPool,5,com.company.testautomation.CucumberRunner] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[OkHttp ConnectionPool,5,com.company.testautomation.CucumberRunner] will linger despite being asked to die via interruption
[WARNING] thread Thread[Okio Watchdog,5,com.company.testautomation.CucumberRunner] will linger despite being asked to die via interruption
[WARNING] thread Thread[UrlChecker-19,5,com.company.testautomation.CucumberRunner] will linger despite being asked to die via interruption
[WARNING] NOTE: 3 thread(s) did not finish despite being asked to  via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=com.company.testautomation.CucumberRunner,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:776)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:321)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:29 h
[INFO] Finished at: 2019-11-29T00:17:20+03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (Execute Cucumber tests) on project TestAutomation: An exception occured while executing the Java class. org.apache.http.conn.HttpHostConnectException: Connect to localhost:23723 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
[ERROR] Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
[ERROR] System info: host: 'ip-XcensoredX', ip: 'XcensoredX', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-1051-aws', java.version: '11.0.4'
[ERROR] Driver info: driver.version: RemoteWebDriver
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

And build fails. If a pipeline works, good. If it doesn't, then it needs a fix. But having an unstable pipeline is something that nobody wants.

Thanks in advance.

Answer 1

Add

<properties>
    <java.version>...</java.version>
    <exec.mainClass>....<mainClass>
    <exec.cleanupDaemonThreads>false</exec.cleanupDaemonThreads>
</properties>

To your pom.xml or add

-Dexec.cleanupDaemonThreads=false

to the command line.