Remote agents get 404 error when attempting to get classpath info from server

We attempted an upgrade of bamboo from v. 5.9.7 to 6.2.5. Server installation was successful, but the remote agents failed to start up. We get the following agent log entries:


INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,262 INFO [WrapperSimpleAppMain] [ClasspathBuilder] Syncing /1fb/home/build/bamboo-agent-home with https://ci-server/agentServer/bootstrap/classpath/manifest
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,276 ERROR [WrapperSimpleAppMain] [ClasspathBuilder] An error has occurred when retrieving classpath data: HTTP/1.1 404 Not Found
INFO | jvm 6 | 2017/12/22 14:47:03 | Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 6 | 2017/12/22 14:47:03 | 2017-12-22 14:47:03,277 FATAL [WrapperSimpleAppMain] [AgentBootstrap] Exiting due to fatal exception.
INFO | jvm 6 | 2017/12/22 14:47:03 | com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: Response: [HTTP/1.1 404 Not Found] received after manifest retrieval request.
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.terminateWithError(ClasspathBuilder.java:211)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getJarList(ClasspathBuilder.java:200)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.sync(ClasspathBuilder.java:85)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.ClasspathBuilder.getClassLoader(ClasspathBuilder.java:58)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.newClassLoader(AgentContext.java:329)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:93)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:111)
INFO | jvm 6 | 2017/12/22 14:47:03 | at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.main(AgentBootstrap.java:50)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 6 | 2017/12/22 14:47:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 6 | 2017/12/22 14:47:03 | at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO | jvm 6 | 2017/12/22 14:47:03 | at java.lang.Thread.run(Thread.java:748)
FATAL | wrapper | 2017/12/22 14:47:03 | There were 5 failed launches in a row, each lasting less than 3 seconds. Giving up.
FATAL | wrapper | 2017/12/22 14:47:03 | There may be a configuration problem: please check the logs.
STATUS | wrapper | 2017/12/22 14:47:03 | <-- Wrapper Stopped

 

The agent is trying to reach the server URL https://ci-server/agentServer/bootstrap/classpath/manifest and is receiving a 404 error in return.  

 

Entering the above URL directly into a browser also yields a 404 error as expected.

There are no apparent server log entries at the time of the agent request, or when we entered the URL into a browser manually, but we did find this error on the server log at a previous time, which appears related.

 

2017-12-22 13:50:09,838 ERROR [ajp-nio-8809-exec-30] [FiveOhOh] 500 Exception was thrown.
javax.servlet.ServletException: Error instantiating servlet class com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1120)
at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:828)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:135)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:873)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:670)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.apache.http.entity.ContentType.withCharset(Ljava/nio/charset/Charset;)Lorg/apache/http/entity/ContentType;
at com.atlassian.bamboo.agent.classserver.cacheable.CacheableClasspathServlet.<clinit>(CacheableClasspathServlet.java:56)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:119)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1102)
... 17 more

 

These logs indicate that the JVM couldn't find a method withCharset(Charset c) in the class org.apache.http.entity.ContentType. 

As far as I can tell, the above referenced class (org.apache.http.entity.ContentType) is in httpcore.4.4.6.jar in the WEB-INF/lib folder of the bamboo installation. It appears that this jar file does include a version of a ContentType class with method withCharset(Charset c) that returns a ContentType, according to my decompiler.   Also, there doesn't seem to be any other version of this library in the classpath anywhere.

 

Any thoughts as to how to fix this issue?

 

We have since rolled back to version 5.9.7 but are anxious to get 6.2.5 working due to security vulnerabilities in 5.9.7.

2 answers

This widget could not be displayed.

Hi Peter,

Make sure you have checked the supported platforms.

https://confluence.atlassian.com/display/BAMBOO0602/Supported+platforms

Try a workaround here, based on the same error

https://jira.atlassian.com/browse/BAM-18279

Or you could re-install the remote agent using the 6.2.5 jar

This widget could not be displayed.

By using verbose classloader logging, we were able to find out where the incorrect version of the org.apache.http.entity.ContentType class came from. It came from bamboo-artifactory-plugin-1.8.2.jar which somehow ended up in atlassian-bamboo/WEB-INF/lib. I am not sure how this jar file ended up in that directory, as it's not in the clean downloaded version of  bamboo 6.2.5 jar file.  But I confirmed this jar file contains an old version of http-core from apache.

 

We renamed this jar file (to prevent its loading into the JVM).  We restarted the server, and this seemed to fix the problem on the server side, or at least we didn't encounter this exception subsequently.

 

However, when we restarted the remote agent, we got a similar exception

 

Constructor threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: com.google.common.base.Equivalence.identity()Lcom/google/common/base/Equivalence;

 

Again, it seems that the artifactory plugin is the root of the problem.  We found bamboo-artifactory-1.8.2.jar in the agent install location, in the classpath directory.  This jar file also contains an older version of the google guava libraries. 

 

So I think our mystery is solved - an old version of the plugin is the cause of our issues.

Lesson learned - reinstall up-to-date plugins when upgrading Bamboo.

 

FYI - it appears the current version of the Artifactory plugin is 2.3.0. We will try again with this version.

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Posted Wednesday in New to Jira

Are you planning to trial, or are currently trialling Jira Software? - We want to talk to you!

Hello! I'm Rayen, a product manager at Atlassian. My team and I are working hard to improve the trial experience for Jira Software Cloud. We are interested in   talking to 20 people planning t...

83 views 1 0
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you