Remote agents get 404 error when attempting to get classpath info from server

[Peter Stelzer]

We attempted an upgrade of bamboo from v. 5.9.7 to 6.2.5. Server installation was successful, but the remote agents failed to start up. We get the following agent log entries:

INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,262 INFO [WrapperSimpleAppMain] [ClasspathBuilder] Syncing /1fb/home/build/bamboo-agent-home with https://ci-server/agentServer/bootstrap/classpath/manifest
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,276 ERROR [WrapperSimpleAppMain] [ClasspathBuilder] An error has occurred when retrieving classpath data: HTTP/1.1 404 Not Found
INFO | jvm 6 | 2017/12/22 14:47:03 | Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,277 FATAL [WrapperSimpleAppMain] [AgentBootstrap] Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
FATAL | wrapper | 2017/12/22 14:47:03 | There were 5 failed launches in a row, each lasting less than 3 seconds. Giving up.
FATAL | wrapper | 2017/12/22 14:47:03 | There may be a configuration problem: please check the logs.
STATUS | wrapper | 2017/12/22 14:47:03 | <-- Wrapper Stopped

The agent is trying to reach the server URL https://ci-server/agentServer/bootstrap/classpath/manifest and is receiving a 404 error in return.  

Entering the above URL directly into a browser also yields a 404 error as expected.

There are no apparent server log entries at the time of the agent request, or when we entered the URL into a browser manually, but we did find this error on the server log at a previous time, which appears related.

2017-12-22 13:50:09,838 ERROR [ajp-nio-8809-exec-30] [FiveOhOh] 500 Exception was thrown.
javax.servlet.ServletException: Error instantiating servlet class com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1120)
at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:828)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:135)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:873)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:670)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.apache.http.entity.ContentType.withCharset(Ljava/nio/charset/Charset;)Lorg/apache/http/entity/ContentType;
at com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet.<clinit>(CacheableClasspathServlet.java:56)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:119)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1102)
... 17 more

These logs indicate that the JVM couldn't find a method withCharset(Charset c) in the class org.apache.http.entity.ContentType. 

As far as I can tell, the above referenced class (org.apache.http.entity.ContentType) is in httpcore.4.4.6.jar in the WEB-INF/lib folder of the bamboo installation. It appears that this jar file does include a version of a ContentType class with method withCharset(Charset c) that returns a ContentType, according to my decompiler.   Also, there doesn't seem to be any other version of this library in the classpath anywhere.

Any thoughts as to how to fix this issue?

We have since rolled back to version 5.9.7 but are anxious to get 6.2.5 working due to security vulnerabilities in 5.9.7.

[Peter Stelzer]

By using verbose classloader logging, we were able to find out where the incorrect version of the org.apache.http.entity.ContentType class came from. It came from bamboo-artifactory-plugin-1.8.2.jar which somehow ended up in atlassian-bamboo/WEB-INF/lib. I am not sure how this jar file ended up in that directory, as it's not in the clean downloaded version of  bamboo 6.2.5 jar file.  But I confirmed this jar file contains an old version of http-core from apache.

We renamed this jar file (to prevent its loading into the JVM).  We restarted the server, and this seemed to fix the problem on the server side, or at least we didn't encounter this exception subsequently.

However, when we restarted the remote agent, we got a similar exception

Constructor threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: com.google.common.base.Equivalence.identity()Lcom/google/common/base/Equivalence;

Again, it seems that the artifactory plugin is the root of the problem.  We found bamboo-artifactory-1.8.2.jar in the agent install location, in the classpath directory.  This jar file also contains an older version of the google guava libraries. 

So I think our mystery is solved - an old version of the plugin is the cause of our issues.

Lesson learned - reinstall up-to-date plugins when upgrading Bamboo.

FYI - it appears the current version of the Artifactory plugin is 2.3.0. We will try again with this version.

[edwin]

Hi Peter,

Make sure you have checked the supported platforms.

https://confluence.atlassian.com/display/BAMBOO0602/Supported+platforms

Try a workaround here, based on the same error

https://jira.atlassian.com/browse/BAM-18279

Or you could re-install the remote agent using the 6.2.5 jar