Remote agents get 404 error when attempting to get classpath info from server

We attempted an upgrade of bamboo from v. 5.9.7 to 6.2.5. Server installation was successful, but the remote agents failed to start up. We get the following agent log entries:


INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,262 INFO [WrapperSimpleAppMain] [ClasspathBuilder] Syncing /1fb/home/build/bamboo-agent-home with https://ci-server/agentServer/bootstrap/classpath/manifest
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,276 ERROR [WrapperSimpleAppMain] [ClasspathBuilder] An error has occurred when retrieving classpath data: HTTP/1.1 404 Not Found
INFO | jvm 6 | 2017/12/22 14:47:03 | Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,277 FATAL [WrapperSimpleAppMain] [AgentBootstrap] Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
FATAL | wrapper | 2017/12/22 14:47:03 | There were 5 failed launches in a row, each lasting less than 3 seconds. Giving up.
FATAL | wrapper | 2017/12/22 14:47:03 | There may be a configuration problem: please check the logs.
STATUS | wrapper | 2017/12/22 14:47:03 | <-- Wrapper Stopped

 

The agent is trying to reach the server URL https://ci-server/agentServer/bootstrap/classpath/manifest and is receiving a 404 error in return.  

 

Entering the above URL directly into a browser also yields a 404 error as expected.

There are no apparent server log entries at the time of the agent request, or when we entered the URL into a browser manually, but we did find this error on the server log at a previous time, which appears related.

 

2017-12-22 13:50:09,838 ERROR [ajp-nio-8809-exec-30] [FiveOhOh] 500 Exception was thrown.
javax.servlet.ServletException: Error instantiating servlet class com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1120)
at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:828)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:135)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:873)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:670)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.apache.http.entity.ContentType.withCharset(Ljava/nio/charset/Charset;)Lorg/apache/http/entity/ContentType;
at com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet.<clinit>(CacheableClasspathServlet.java:56)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:119)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1102)
... 17 more

 

These logs indicate that the JVM couldn't find a method withCharset(Charset c) in the class org.apache.http.entity.ContentType. 

As far as I can tell, the above referenced class (org.apache.http.entity.ContentType) is in httpcore.4.4.6.jar in the WEB-INF/lib folder of the bamboo installation. It appears that this jar file does include a version of a ContentType class with method withCharset(Charset c) that returns a ContentType, according to my decompiler.   Also, there doesn't seem to be any other version of this library in the classpath anywhere.

 

Any thoughts as to how to fix this issue?

 

We have since rolled back to version 5.9.7 but are anxious to get 6.2.5 working due to security vulnerabilities in 5.9.7.

2 answers

Hi Peter,

Make sure you have checked the supported platforms.

https://confluence.atlassian.com/display/BAMBOO0602/Supported+platforms

Try a workaround here, based on the same error

https://jira.atlassian.com/browse/BAM-18279

Or you could re-install the remote agent using the 6.2.5 jar

By using verbose classloader logging, we were able to find out where the incorrect version of the org.apache.http.entity.ContentType class came from. It came from bamboo-artifactory-plugin-1.8.2.jar which somehow ended up in atlassian-bamboo/WEB-INF/lib. I am not sure how this jar file ended up in that directory, as it's not in the clean downloaded version of  bamboo 6.2.5 jar file.  But I confirmed this jar file contains an old version of http-core from apache.

 

We renamed this jar file (to prevent its loading into the JVM).  We restarted the server, and this seemed to fix the problem on the server side, or at least we didn't encounter this exception subsequently.

 

However, when we restarted the remote agent, we got a similar exception

 

Constructor threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: com.google.common.base.Equivalence.identity()Lcom/google/common/base/Equivalence;

 

Again, it seems that the artifactory plugin is the root of the problem.  We found bamboo-artifactory-1.8.2.jar in the agent install location, in the classpath directory.  This jar file also contains an older version of the google guava libraries. 

 

So I think our mystery is solved - an old version of the plugin is the cause of our issues.

Lesson learned - reinstall up-to-date plugins when upgrading Bamboo.

 

FYI - it appears the current version of the Artifactory plugin is 2.3.0. We will try again with this version.

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Monday in Jira Ops

Jira Ops Early Access Program Update #1: Announcing our next feature and a new integration

Thanks for signing up for Jira Ops! I’m Matt Ryall, leader for the Jira Ops product team at Atlassian. Since this is a brand new product, we’ll be delivering improvements quickly and sharing updates...

496 views 0 9
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you