Bamboo: More than one agent per EC2 elastic instance?

Question

Is it possible to run more than one Bamboo Agent per EC2 Elastic Instance?

We use Docker in our build system and the process seems mostly network IO bound. It would be nice if we could run multiple Agents on one machine.

By running multiple agents on one machine instead of starting multiple machines, we also don't need to worry about shipping Docker images between machines as artifacts.

Update 30 Oct 2014:

I tried copying the current startup script and adding a new home:

$ cat /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent2
#!/bin/sh

bambooAgentBin=$(cd -P -- $(dirname $0) && pwd)
. $bambooAgentBin/bambooShellFunctions.sh

echo Starting Elastic Bamboo Agent...
java -Dbamboo.home=/home/bamboo/bamboo-agent-home-2/ -Dimagefiles.version=3.3-SNAPSHOT -jar $bambooAgentBin/*installer*.jar 2>&1 | tee -a $(getHomeDir)/bamboo-elastic-agent.out

It seems like the elastic version sets up some kind of a tunnel and crashes because a tunnel is already running:

# su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent2 - bamboo  &
[1] 14143
Starting Elastic Bamboo Agent...
2014-10-30 08:39:31,804 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/ to /opt/bamboo-elastic-agent
2014-10-30 08:39:31,979 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/ to /opt/bamboo-elastic-agent
2014-10-30 08:39:31,979 INFO [main] [S3Utils] Fetching the list of remote objects...
2014-10-30 08:39:33,006 INFO [main] [S3Utils] Found 579 files in s3://bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/
2014-10-30 08:39:33,059 INFO [main] [S3Utils] Found 463 files in /opt/bamboo-elastic-agent
2014-10-30 08:39:33,060 INFO [main] [S3Utils] Generating the list of files to fetch from S3...
2014-10-30 08:39:33,076 INFO [main] [S3Utils] Generating the list of files to remove...
2014-10-30 08:39:33,078 INFO [main] [S3Utils] Removing 0 files from /opt/bamboo-elastic-agent
2014-10-30 08:39:33,079 INFO [main] [S3Utils] Fetching 155 files to /opt/bamboo-elastic-agent
2014-10-30 08:39:39,969 INFO [main] [S3Utils] Fetched 113 MB from S3
2014-10-30 08:39:39,973 INFO [main] [ElasticAgentInstaller] Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -Dimagefiles.version=3.3, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, -cp, /opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-io-2.4.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-0.21.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/guava-bridge-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.8.jar:/opt/bamboo-elastic-agent/boot/atlassian-util-concurrent-2.4.1.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.15.jar:/opt/bamboo-elastic-agent/boot/guava-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.71.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.1.1.jar:/opt/bamboo-elastic-agent/boot/fugue-1.1.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.7.1.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.2.5.jar:/opt/bamboo-elastic-agent/boot/commons-logging-1.0.4.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.1.1.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.4.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.2.5.jar:, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap]
2014-10-30 08:39:40,119 INFO [main] [ElasticAgentBootstrap] Starting Agent Bootstrap using Java 1.6.0_45 from Sun Microsystems Inc.
2014-10-30 08:39:40,410 INFO [main] [ElasticAgentBootstrap] Using tunnelling. Setting virtual host name to https://xxxxxxx.atlassian.net/builds/agentServer/
2014-10-30 08:39:40,410 INFO [main] [ElasticAgentBootstrap] Using tunnelling for HTTP(S). Registering 'httpt' and 'httpst' protocols.
2014-10-30 08:39:40,416 INFO [main] [ElasticAgentBootstrap] HTTP(S) tunnel: enabled
2014-10-30 08:39:40,416 INFO [main] [ElasticAgentBootstrap] JMS tunnel: enabled
2014-10-30 08:39:40,424 INFO [main] [ElasticAgentBootstrap] Starting tunnel server, waiting for 2 connections.
2014-10-30 08:39:40,425 FATAL [tunnellogger-thread] [TunnelServer] [com.atlassian.tunnel.tunnel.server.TunnelServer] Fatal error in TunnelServer.
java.net.BindException: Address already in use
    at java.net.PlainSocketImpl.socketBind(Native Method)
    at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
    at java.net.ServerSocket.bind(ServerSocket.java:328)
    at java.net.ServerSocket.<init>(ServerSocket.java:194)
    at java.net.ServerSocket.<init>(ServerSocket.java:150)
    at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:84)
    at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:81)
    at com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:58)
    at com.atlassian.tunnel.tunnel.server.TunnelServer.run(TunnelServer.java:54)
    at java.lang.Thread.run(Thread.java:662)

Any idea for a workaround?

Answer 1

We (Atlassian Build Engineering) have created a set of plugins to run a number Docker based agents in a cluster (ECS) that come online, build a single job and then exit. It should be able to do what you outlined. We've recently open sourced the solution.

See https://bitbucket.org/atlassian/per-build-container for more details.

Answer 2

Amazon instances are historically bad providing I/O capacity (I am assuming you mean network I/O) per dollar as they tend to be optimized more for CPU and Memory intensive workloads. You may find that additional processes on the same node may not help.

If you install your docker image on a node and take a snapshot to create an AMI after the image is installed, you can launch as many ec2 instances with that AMI as you want. They will have the image preinstalled.

You can also use CloudFormation and/or Cloud Init to recompile or download your image on each Ec2 instance without having to worry about manually moving around docker image.

However, if you do want to run more than one bamboo agent on a node you should be able too as longs as you set the bamboo.home parameter differently for each agent instance.

java -Dbamboo.home=/agent1Home -jar atlassian-bamboo-agent-installer-X.X-SNAPSHOT.jar \
       http://bamboo-host-server:8085/agentServer/
java -Dbamboo.home=/agent2Home -jar atlassian-bamboo-agent-installer-X.X-SNAPSHOT.jar \
       http://bamboo-host-server:8085/agentServer/