Remote Agent Installation on AWS EC2 fails

[Jeremy Roelfs]

After following this tutorial to install Remote Agent on Windows.

I cannot successfully install the Agent.

I think the problem is when it writes the classpath file(s) back the the Bamboo Server it is failing.

We have opened all ports on both our remote EC2 and our Bamboo Server EC2, but nothing works. We have allowed all modify/read/write in our folder security settings on both EC2s.

Here is my log:

 22:49:28 | 2018-01-24 22:49:27,963 INFO [WrapperSimpleAppMain] [AgentBootstrap] Starting Agent Bootstrap using Java 1.8.0_161 from Oracle Corporation. Default charset: windows-1252, file name encoding: Cp1252
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:27,963 INFO [WrapperSimpleAppMain] [AgentBootstrap] Agent bootstrap using endpoint: http://***/agentServer/installntservice
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:27,978 INFO [WrapperSimpleAppMain] [RemoteAgentHomeLocatorForBootstrap] Using agent home located at [C:\bamboo-agent]
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:27,978 INFO [WrapperSimpleAppMain] [RemoteAgentHomeLocatorForBootstrap] Using agent home located at [C:\bamboo-agent]
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:27,978 INFO [WrapperSimpleAppMain] [AgentUuidInitializer] Found agent UUID '*****' in temporary UUID file 'C:\bamboo-agent\uuid-temp.properties'
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:27,978 INFO [WrapperSimpleAppMain] [AgentContext] Requesting fingerprint, url: *****/installntservice/GetFingerprint.action?hostName=********&version=3&agentUuid=************
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:27,994 INFO [WrapperSimpleAppMain] [RemoteAgentHttpClientFactory] Creating a new HTTP client...
INFO | jvm 2 | 2018/01/24 22:49:28 | 2018-01-24 22:49:28,900 INFO [WrapperSimpleAppMain] [AgentContext] Authenticating against Bamboo server at *********/agentServer/installntservice/
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,010 INFO [WrapperSimpleAppMain] [AgentContext] Successfully authenticated with Bamboo server
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,010 INFO [WrapperSimpleAppMain] [BootstrapUpdater] Checking if bootstrap C:\bamboo-agent\lib\bamboo-agent-bootstrap.jar is up to date, using url: ************/installntservice/GetJar?name=bootstrap-with-deps&checksum=********&fingerprint=*********************
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,056 WARN [WrapperSimpleAppMain] [BootstrapUpdater] Unable to update bootstrap: HTTP/1.1 404 Not Found, continuing with the existing bootstrap.
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,056 INFO [WrapperSimpleAppMain] [AgentContext] Bootstrap Type: [REMOTE], agent class: com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,056 INFO [WrapperSimpleAppMain] [AgentContext] Preparing for a classloader switch
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,056 INFO [WrapperSimpleAppMain] [RemoteAgentHomeLocatorForBootstrap] Using agent home located at [C:\bamboo-agent]
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,056 INFO [WrapperSimpleAppMain] [ClasspathBuilder] Syncing [C:\bamboo-agent\classpath] with **********/agentServer/installntservice/classpath.zip?fingerprint=*************
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,213 INFO [WrapperSimpleAppMain] [ClasspathBuilder] 334 file(s) currently in [C:\bamboo-agent\classpath]
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,260 ERROR [WrapperSimpleAppMain] [ClasspathBuilder] An error has occurred when retrieving classpath data: HTTP/1.1 404 Not Found
INFO | jvm 2 | 2018/01/24 22:49:29 | Exiting due to fatal exception.
INFO | jvm 2 | 2018/01/24 22:49:29 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after 'classpath.zip' retrieval request.
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getRemoteResources(ClasspathBuilder.java:118)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.build(ClasspathBuilder.java:59)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:388)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:98)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:117)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:52)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 2 | 2018/01/24 22:49:29 | at java.lang.Thread.run(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | 2018-01-24 22:49:29,260 FATAL [WrapperSimpleAppMain] [AgentBootstrap] Exiting due to fatal exception.
INFO | jvm 2 | 2018/01/24 22:49:29 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after 'classpath.zip' retrieval request.
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getRemoteResources(ClasspathBuilder.java:118)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.build(ClasspathBuilder.java:59)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:388)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:98)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:117)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:52)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 2 | 2018/01/24 22:49:29 | at java.lang.Thread.run(Unknown Source)
ERROR | wrapper | 2018/01/24 22:49:29 | JVM exited while starting the application.
STATUS | wrapper | 2018/01/24 22:49:33 | Launching a JVM...
INFO | jvm 3 | 2018/01/24 22:49:34 | Wrapper (Version 3.2.3-atlassian-1) http://wrapper.tanukisoftware.org
INFO | jvm 3 | 2018/01/24 22:49:34 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.

[somethingblue]

Hi Jeremy,

Hopefully you've resolved this by now but if not I may have found a bug that may help. 

The following portion of your stacktrace matches Remote agents do not work with HTTP to HTTPS redirects in Tomcat web.xml:

INFO | jvm 2 | 2018/01/24 22:49:29 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after 'classpath.zip' retrieval request.
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getRemoteResources(ClasspathBuilder.java:118)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.build(ClasspathBuilder.java:59)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:388)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:98)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:117)
INFO | jvm 2 | 2018/01/24 22:49:29 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:52)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 2 | 2018/01/24 22:49:29 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)

The bug is titled and there is a workaround listed:

1. In the remote agent edit the "wrapper.app.parameter" in <agent-home>/conf/wrapper.conf and change the protocol to https.
   E.g: wrapper.app.parameter=http://bamboo-server.com/agentServer/ change to wrapper.app.parameter=https://bamboo-server.com/agentServer/
2. Restart the remote agent

Try that and let me know if it helps.  If so please vote on BAM-18279 to add impact and you'll be notified of any updates to the ticket.

Cheers,

Branden

[Jeremy Roelfs]

I did try this and it did not work.
I am thinking my problem is actually the Java version. I used 8 and received the above errors. When I used version 9, I received different errors. Java 7 is unavailable. Maybe the agent doesn't work with 8 or 9?

[somethingblue]

Hi Jeremy,

If you take a look at the agent logs do you see any networking errors?

[Jeremy Roelfs]

We have terminated that instance, so I can't look at the logs anymore. But I do not remember seeing any thing. Just incase there were any port issues, so we allowed All Traffic on the EC2 and on the Bamboo Server EC2. We also allowed all permissions for the directories.

None of that seemed to make a difference.

[somethingblue]

Once you restart the instance check to see if you find any network related issues.  Let me know either way and we'll go from there.

In regards to the JAVA version, there is a bug related to that, however, it states Java 8 is required and Java 6/7 does not work.  

Let me know what you find.

[Jeremy Roelfs]

@somethingblue,

I found another error. I ran the install a few different ways and I came up with an SSL error. I tried the https://, but that did not work so I'm following this to see if that is my issue.

My end goal is to call an executable on that EC2 instance from my Bamboo Build Plan.

Do I need to have Bamboo Server and GIT installed on that remote EC2 instance?

[somethingblue]

Hi Jeremy,

Can you paste the SSL error you see in the Catalina.out?

[Jeremy Roelfs]

@somethingblueI was unable to get the information for you Branden.  I decided to switch to AWS Run Command to invoke the tests on the EC2 from a Bamboo Script. I could not spend any more time debugging the remote agent. I am not sure if it is the SSL or how we have our system setup.