Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Products
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

New Elastic Bamboo agent stuck in "Pending", failing at initFingerprint()

Ian Su
Ian Su April 15, 2013

We are using Bamboo version 3.4.3.

I'm trying to set up Elastic Bamboo, but when I start an agent instance (using a t1.micro of AMI Default Image EBS i386 (linux)), it gets stuck in "Pending" state after the EC2 instance comes up.

I am able to SSH into the EC2 instance and /home/bamboo/bamboo-elastic-agent.out reads:

0 [main] INFO com.atlassian.bamboo.agent.elastic.S3Sync  - Syncing from: bamboo-agent-release-us-e1/3.4.3/743f2e6a4814/ to /opt/bamboo-elastic-agent
146 [main] INFO com.atlassian.aws.s3.S3Utils  - Syncing s3://bamboo-agent-release-us-e1/3.4.3/743f2e6a4814/ to /opt/bamboo-elastic-agent
147 [main] INFO com.atlassian.aws.s3.S3Utils  - Fetching the list of remote objects...
95993 [main] INFO com.atlassian.aws.s3.S3Utils  - Removing 894 files from /opt/bamboo-elastic-agent
96148 [main] INFO com.atlassian.aws.s3.S3Utils  - Fetching 104 files to /opt/bamboo-elastic-agent
108661 [main] INFO com.atlassian.aws.s3.S3Utils  - Fetched 57 MB from S3
108663 [main] INFO com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller  - Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -cp, /opt/bamboo-elastic-agent/boot/atlassian-util-concurrent-2.3.0.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-3.4.3.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.14.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-0.15.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-3.4.3.jar:/opt/bamboo-elastic-agent/boot/commons-httpclient-3.0.1.jar:/opt/bamboo-elastic-agent/boot/commons-io-1.4-backport-IO-168.jar:/opt/bamboo-elastic-agent/boot/gson-1.6.jar:/opt/bamboo-elastic-agent/boot/jaxb-api-2.1.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.4.jar:/opt/bamboo-elastic-agent/boot/typica-1.6.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15-140.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.4.jar:/opt/bamboo-elastic-agent/boot/commons-logging-1.0.4.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.15.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.5.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.1.5.jar:/opt/bamboo-elastic-agent/boot/annotations-6.0.5.jar:/opt/bamboo-elastic-agent/boot/jaxb-impl-2.1.10.jar:/opt/bamboo-elastic-agent/boot/guava-8.1.jar:/opt/bamboo-elastic-agent/boot/activation-1.1.1.jar:/opt/bamboo-elastic-agent/boot/servlet-api-2.5.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-3.4.3.jar:, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap]
1 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  - Starting Agent Bootstrap using Java 1.6.0_26 from Sun Microsystems Inc.
1950 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  - Using tunnelling. Registering 'httpt' and 'httpst' protocols.
1985 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  - Starting tunnel server
1986 [PS AUX and LSOF watchdog] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  - Starting watchdog
2124 [PS AUX and LSOF watchdog] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  - Exit code: 127
2676 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Waiting for tunnel connection.
2677 [tunnellogger-thread] INFO com.atlassian.tunnel.logger.LocklessLogger  - [tunnellogger-thread] Logging system is operating in a lockless mode
12675 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Accepted tunnel connection for port 43570
12675 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Waiting for tunnel connection.
12676 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Accepted tunnel connection for port 10663
12690 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Waiting for tunnel connection.
13746 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelAcceptor  - [tunnelserver-26224-1-thread-2] Will try to accept tunnel by creating a socket for port '4527'
13748 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelAcceptor  - [tunnelserver-26224-1-thread-1] Will try to accept tunnel by creating a socket for port '46593'
13749 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.LocalAcceptor  - [tunnelserver-26224-1-thread-3] Waiting for local connection on port 46593.
Agent bootstrap using baseUrl: httpt://127.0.0.1:46593/agentServer/
13760 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.LocalAcceptor  - [tunnelserver-26224-1-thread-4] Waiting for local connection on port 4527.
13781 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  - Using tunnelling. Setting virtual host name to bamboo.sirca.org.au
14489 [main] INFO com.atlassian.bamboo.agent.bootstrap.RemoteAgentHomeLocatorForBootstrap  - Agent home located at '/home/bamboo/bamboo-agent-home'
14489 [main] INFO com.atlassian.bamboo.agent.bootstrap.RemoteAgentHomeLocatorForBootstrap  - Creating agent home located at '/home/bamboo/bamboo-agent-home'
14490 [main] INFO com.atlassian.bamboo.agent.bootstrap.RemoteAgentHomeLocatorForBootstrap  - Agent home located at '/home/bamboo/bamboo-agent-home'
14491 [main] INFO com.atlassian.bamboo.agent.bootstrap.AgentUuidInitializer  - Generating new UUID for this agent: 227f9aae-ff3d-4a2a-9a7f-71327f22b5e2
Requesting fingerprint, url: httpt://127.0.0.1:46593/agentServer/GetFingerprint.action?hostName=domU-12-31-39-17-2D-0A.compute-1.internal&agentType=elastic&instanceId=i-677f2005&version=3&agentUuid=227f9aae-ff3d-4a2a-9a7f-71327f22b5e2
14526 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.LocalAcceptor  - [tunnelserver-26224-1-thread-3] Accepted local connection on port 46593.
19971 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Accepted tunnel connection for port 13480
19971 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelServer  - [com.atlassian.tunnel.tunnel.server.TunnelServer] Waiting for tunnel connection.
20414 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.TunnelAcceptor  - [tunnelserver-26224-1-thread-3] Will try to accept tunnel by creating a socket for port '46593'
20415 [tunnellogger-thread] INFO com.atlassian.tunnel.tunnel.server.LocalAcceptor  - [tunnelserver-26224-1-thread-5] Waiting for local connection on port 46593.
44550 [main] INFO org.apache.commons.httpclient.HttpMethodDirector  - I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
44550 [main] INFO org.apache.commons.httpclient.HttpMethodDirector  - Retrying request

... last few lines repeats over and over until ...

Exiting due to fatal exception.
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
        at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
        at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1373)
        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
        at com.atlassian.bamboo.agent.bootstrap.AgentContext.initFingerprint(AgentContext.java:127)
        at com.atlassian.bamboo.agent.bootstrap.AgentContext.initServerSession(AgentContext.java:112)
        at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:103)
        at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:82)
        at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.run(ElasticAgentBootstrap.java:96)
        at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.main(ElasticAgentBootstrap.java:57)
875317 [main] FATAL com.atlassian.bamboo.agent.bootstrap.AgentBootstrap  - Exiting due to fatal exception.
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
        at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
        at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1373)
        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
        at com.atlassian.bamboo.agent.bootstrap.AgentContext.initFingerprint(AgentContext.java:127)
        at com.atlassian.bamboo.agent.bootstrap.AgentContext.initServerSession(AgentContext.java:112)
        at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:103)
        at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:82)
        at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.run(ElasticAgentBootstrap.java:96)
        at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.main(ElasticAgentBootstrap.java:57)

So it looks like it is unable to use the tunnel to talk to the bamboo server, at the stage where it is trying to call the initFingerprint() method. However, using netstat and tcpdump, we have determined that the connection is established successfully over port 26224, and there is traffic over that connection. We are at a loss as to what to look into next.

Any advice would be greatly appreciated! Thanks.

Answer
Watch
Like Be the first to like this
864 views

2 answers

1 accepted

0 votes
Answer accepted
Ian Su
Ian Su April 15, 2013

Thanks! That pointed us in the right direction.

There was an iptables NAT rule on the bamboo server that redirected incoming traffic from port 80 to 8085 (where bamboo was actually running). This redirection is not applied when trying to access the host from localhost and so the tunnelling does not work.

Once we fixed the redirection for localhost the elastic agent started working.

Cheers!

View More Comments
Przemek Bruski
Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 15, 2013

Yup, that's what I have suspected.

Like
Reply
2 votes
Przemek Bruski
Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 15, 2013

If you log in to your Bamboo server and from that server try to access Bamboo's base URL, does it work?

View More Comments
EddieW
EddieW
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
December 3, 2013

Hey Przemek. Do you mean login to the bamboo master or agent server?

Our master server has no issues reaching the configured URL, but our remote agents cannot, as it was described that the SSL Tunnel port 36XXX was the only connectivity needed.

Like
Reply

Suggest an answer

Log in or Sign up to answer
Bamboo
Bamboo
TAGS
AUG Leaders

Atlassian Community Events

    See all events