Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,359,800
Community Members
 
Community Events
168
Community Groups

heartbeat/broker timeout, communication issue between server and agents

We currently experiencing a problem on my Bamboo Server 6.9 for heart beat issues show on following remote agent logs, and after it reaches 10 mins timeout, the bamboo server just show 'Agent 'remote-bamboo-agent-01' went offline while building xxxxx'. The results of that build will not be available.'

INFO | jvm 5 | 2020/12/26 02:14:29 | 2020-12-26 02:14:29,652 INFO [scheduler_Worker-4] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 3 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:15:29 | 2020-12-26 02:15:29,651 INFO [scheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 4 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:16:29 | 2020-12-26 02:16:29,652 INFO [scheduler_Worker-9] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 5 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:17:29 | 2020-12-26 02:17:29,652 INFO [scheduler_Worker-2] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 6 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:18:29 | 2020-12-26 02:18:29,653 INFO [scheduler_Worker-3] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 7 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:19:29 | 2020-12-26 02:19:29,652 INFO [scheduler_Worker-5] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 8 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:20:29 | 2020-12-26 02:20:29,652 INFO [scheduler_Worker-7] [AgentHeartBeatJob
..............................
INFO | jvm 5 | 2020/12/26 02:23:29 | 2020-12-26 02:23:29,660 FATAL [scheduler_Worker-10] [AgentContext] Agent encountered a fatal error: 'The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent'. Exiting.
INFO | jvm 5 | 2020/12/26 02:23:29 | java.util.concurrent.TimeoutException: The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent

I have tried the following measures as mentioned on bamboo server documentations:
1. Increase bamboo server heap size to 16GB
2. Delete all build logs in <BAMBOO_HOME>/xml-data/builds
3. Sync bamboo servers and remote agent to same NTP servers

All remote agents are solo instance on 16 cores 32GB machines, and the Atlassian suite is also hosted on an 64 cores machines.

In addition, I have also tried to increase heartbeat timeout intervals in
bamboo server <BAMBOO_HOME>/bin/setenv.sh as following "${JVM_SUPPORT_RECOMMENDED_ARGS:="bamboo.agent.heartbeatTimeoutSeconds=1200"}". And also added 'wrapper.successful_invocation_time=3, wrapper.max_failed_invocations=720
' to remote agent's wrapper.conf .

However, the change of heartbeat timeout settings on bamboo server and remote agents doesn't apply, and the timeout time is till 10mins(600s) by default values.

After several tries to catch the network stats, the TCP send Q both accumulated to around 6000 on both bamboo server and remote agent before previous condition happens. And 8085 http port is not reachable.

We are using bamboos to build 65 mins duration C++ builds with bamboo server + 5 remote agents.

Appreciated if anyone can give me some insight or help

 

0 answers

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bamboo

Bamboo 9.0 is now available

Hey there, Data Center community! I'm Martyna Wojtas and I am the Product Manager for Bamboo Data Center. I’m excited to share that Bamboo 9.0 is now available. We purpose-built this platform to help...

301 views 0 7
Read article

Atlassian Community Events