Remote agents get 404 error when attempting to get classpath info from server

We attempted an upgrade of bamboo from v. 5.9.7 to 6.2.5. Server installation was successful, but the remote agents failed to start up. We get the following agent log entries:


INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,262 INFO [WrapperSimpleAppMain] [ClasspathBuilder] Syncing /1fb/home/build/bamboo-agent-home with https://ci-server/agentServer/bootstrap/classpath/manifest
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,276 ERROR [WrapperSimpleAppMain] [ClasspathBuilder] An error has occurred when retrieving classpath data: HTTP/1.1 404 Not Found
INFO | jvm 6 | 2017/12/22 14:47:03 | Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,277 FATAL [WrapperSimpleAppMain] [AgentBootstrap] Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
FATAL | wrapper | 2017/12/22 14:47:03 | There were 5 failed launches in a row, each lasting less than 3 seconds. Giving up.
FATAL | wrapper | 2017/12/22 14:47:03 | There may be a configuration problem: please check the logs.
STATUS | wrapper | 2017/12/22 14:47:03 | <-- Wrapper Stopped

 

The agent is trying to reach the server URL https://ci-server/agentServer/bootstrap/classpath/manifest and is receiving a 404 error in return.  

 

Entering the above URL directly into a browser also yields a 404 error as expected.

There are no apparent server log entries at the time of the agent request, or when we entered the URL into a browser manually, but we did find this error on the server log at a previous time, which appears related.

 

2017-12-22 13:50:09,838 ERROR [ajp-nio-8809-exec-30] [FiveOhOh] 500 Exception was thrown.
javax.servlet.ServletException: Error instantiating servlet class com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1120)
at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:828)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:135)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:873)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:670)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.apache.http.entity.ContentType.withCharset(Ljava/nio/charset/Charset;)Lorg/apache/http/entity/ContentType;
at com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet.<clinit>(CacheableClasspathServlet.java:56)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:119)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1102)
... 17 more

 

These logs indicate that the JVM couldn't find a method withCharset(Charset c) in the class org.apache.http.entity.ContentType. 

As far as I can tell, the above referenced class (org.apache.http.entity.ContentType) is in httpcore.4.4.6.jar in the WEB-INF/lib folder of the bamboo installation. It appears that this jar file does include a version of a ContentType class with method withCharset(Charset c) that returns a ContentType, according to my decompiler.   Also, there doesn't seem to be any other version of this library in the classpath anywhere.

 

Any thoughts as to how to fix this issue?

 

We have since rolled back to version 5.9.7 but are anxious to get 6.2.5 working due to security vulnerabilities in 5.9.7.

2 answers

Hi Peter,

Make sure you have checked the supported platforms.

https://confluence.atlassian.com/display/BAMBOO0602/Supported+platforms

Try a workaround here, based on the same error

https://jira.atlassian.com/browse/BAM-18279

Or you could re-install the remote agent using the 6.2.5 jar

By using verbose classloader logging, we were able to find out where the incorrect version of the org.apache.http.entity.ContentType class came from. It came from bamboo-artifactory-plugin-1.8.2.jar which somehow ended up in atlassian-bamboo/WEB-INF/lib. I am not sure how this jar file ended up in that directory, as it's not in the clean downloaded version of  bamboo 6.2.5 jar file.  But I confirmed this jar file contains an old version of http-core from apache.

 

We renamed this jar file (to prevent its loading into the JVM).  We restarted the server, and this seemed to fix the problem on the server side, or at least we didn't encounter this exception subsequently.

 

However, when we restarted the remote agent, we got a similar exception

 

Constructor threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: com.google.common.base.Equivalence.identity()Lcom/google/common/base/Equivalence;

 

Again, it seems that the artifactory plugin is the root of the problem.  We found bamboo-artifactory-1.8.2.jar in the agent install location, in the classpath directory.  This jar file also contains an older version of the google guava libraries. 

 

So I think our mystery is solved - an old version of the plugin is the cause of our issues.

Lesson learned - reinstall up-to-date plugins when upgrading Bamboo.

 

FYI - it appears the current version of the Artifactory plugin is 2.3.0. We will try again with this version.

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published May 18, 2017 in Bamboo

FAQ: How to Upgrade Bamboo Server

Bamboo 5.9 will no longer be supported after June 12, 2017. What does this mean? As part of our End of Life policy, Atlassian supports major versions for two years after the first major iteratio...

1,561 views 0 6
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you