After upgrade to Bamboo 6.8, remote agent unable to connect

[sators]

After upgrading from 6.6 to 6.8, my remote agent is unable to connect to the server.  The agent log in the bamboo UI just lists that the remote agent is loading over and over.

Agent atlassian-bamboo log:

2019-02-26 13:06:14,015 WARN [AgentRunnerThread] [FailoverTransport] Transport (tcp://<domain omitted>:54663?wireFormat.maxInactivityDuration=300000) failed , attempting to automatically reconnect: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.

Catalina.out

2019-02-26 19:08:48,409 WARN [http-nio-8085-exec-14] [AuthorizationLoggerListener] Authorization failed: org.acegisecurity.AccessDeniedException: Access is denied; authenticated principal: org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken@9055e4a6: Username: anonymousUser; Password: [PROTECTED]; Authenticated: true; Details: org.acegisecurity.ui.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: ROLE_ANONYMOUS; secure object: com.atlassian.bamboo.configuration.agent.AddRemoteAgent@1e08960; configuration attributes: [WW_ADMIN, WW_RESTRICTED_ADMIN, GLOBAL_READ]

After seeing this authorization failure, I attempted to revoke access to the agent, then restarted the agent.  Upon restarting the agent, it was listed in the Agent Authorization tab pending authorization.  After approving the agent, it still fails to connect with the above "peer did not send his wire format" message.

Now, the Catalina.out messages just say:

2019-02-26 19:20:41,379 INFO [http-nio-8085-exec-12] [RemoteAgentManagerImpl] A remote agent is loading on <domain omitted> (127.0.0.1).

It should be noted that I am using a reverse proxy via Apache which is why the IP address is listed as 127.0.0.1.  However, I have 127.0.0.1 listed as an approved IP address for the agent.

[Foong · Accepted solution]

From the remote agent machine, try running the following command to test the Broker Client URL:

telnet <domain omitted> 54663

Run the following command in the machine hosting Bamboo to check the difference:

telnet localhost 54663

[sators]

Both look the same, just with the IP address different:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

[Foong]

Does it have the following example message?

telnet localhost 54663
Trying ::1...
Connected to localhost.
Escape character is '^]'.
?ActiveMQ
         p
          TcpNoDelayEnabledSizePrefixDisabled CacheSize
                                                         ProviderName  ActiveMQStackTraceEnabledPlatformDetails KJVM: 1.8.0_73, 25.73-b02, Oracle Corporation, OS: Mac OS X, 10.12.6, x86_64
                            CacheEnabledTightEncodingEnabled
                                                            MaxFrameSize???MaxInactivityDuration?? MaxInactivityDurationInitalDelay'ProviderVersion 5.14.5

[sators]

No, nothing is returned after the initial connection.  Screenshot below.  For clarity sake, this command is being run directly on the server running Bamboo.  No characters are being typed other than the initial command.

Trying to manually type `?ActiveMQ` after connected also yields no results/response.

Thank you for your help.

[Foong]

Could it be antivirus blocking the output?

[sators]

Antivirus on the Ubuntu Server?  This was working when Bamboo was on 6.6...the server is pretty vanilla / straight out of the box with Bamboo loaded up and dedicated for its purpose...no antivirus has been loaded/added.

[Foong]

Is the port used by Bamboo Java application?

Run the following command:

lsof -i -P -n | grep 54663

From the result, run the following command to get more detail on the process that have (LISTEN):

ps -p <ID in the second column>

[sators]

Below you can see the results of the output (FYI these commands needed to be run via root in order to see the output).  Note, I am running bamboo as the user "bamboo" as noted in the 3rd column...don't know if that makes any difference?

Also for clarity sake, running `ufw status` confirms that the Ubuntu firewall status is inactive.

[Foong]

During startup, does the log atlassian-bamboo.log shows the following message?

2019-03-06 22:03:30,577 INFO [localhost-startStop-1] [BambooBrokerService] Starting the primary JMS connector...
2019-03-06 22:03:35,599 INFO [localhost-startStop-1] [TransportServerThreadSupport] Listening for connections at: nio://hostname:54663?wireFormat.maxInactivityDuration=300000
2019-03-06 22:03:35,599 INFO [localhost-startStop-1] [TransportConnector] Connector nio://hostname:54663?wireFormat.maxInactivityDuration=300000 started

What is the Broker URL and Broker Client URL setting at Administration > System > General configuration ?

[sators]

Excellent - that lead me to it!  Error messages were in the ~/log/atlassian-bamboo.log in the saying the address unable to be bound - address already in use.  Discovered that bamboo had also been running as a service - so I killed that.

Finally, I had to change the broker URL to not contain the domain name, but be 0.0.0.0.

Thanks for the efforts and helping to lead me down the right path!

[Joel Mokoena]

Hi can you try online to find out what does it take for you to over come that situation

[sators]

As an update - I have tried bypassing the Apache proxy - the Bamboo UI logs now display that the remote agent is loading on the correct source IP address instead of 127.0.0.1, however, the agent still doesn't connect and it's local logs still show the entry for the "peer did not send his wire format".