Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

heartbeat/broker timeout, communication issue between server and agents

We currently experiencing a problem on my Bamboo Server 6.9 for heart beat issues show on following remote agent logs, and after it reaches 10 mins timeout, the bamboo server just show 'Agent 'remote-bamboo-agent-01' went offline while building xxxxx'. The results of that build will not be available.'

INFO | jvm 5 | 2020/12/26 02:14:29 | 2020-12-26 02:14:29,652 INFO [scheduler_Worker-4] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 3 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:15:29 | 2020-12-26 02:15:29,651 INFO [scheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 4 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:16:29 | 2020-12-26 02:16:29,652 INFO [scheduler_Worker-9] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 5 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:17:29 | 2020-12-26 02:17:29,652 INFO [scheduler_Worker-2] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 6 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:18:29 | 2020-12-26 02:18:29,653 INFO [scheduler_Worker-3] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 7 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:19:29 | 2020-12-26 02:19:29,652 INFO [scheduler_Worker-5] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 8 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:20:29 | 2020-12-26 02:20:29,652 INFO [scheduler_Worker-7] [AgentHeartBeatJob
..............................
INFO | jvm 5 | 2020/12/26 02:23:29 | 2020-12-26 02:23:29,660 FATAL [scheduler_Worker-10] [AgentContext] Agent encountered a fatal error: 'The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent'. Exiting.
INFO | jvm 5 | 2020/12/26 02:23:29 | java.util.concurrent.TimeoutException: The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent

I have tried the following measures as mentioned on bamboo server documentations:
1. Increase bamboo server heap size to 16GB
2. Delete all build logs in <BAMBOO_HOME>/xml-data/builds
3. Sync bamboo servers and remote agent to same NTP servers

All remote agents are solo instance on 16 cores 32GB machines, and the Atlassian suite is also hosted on an 64 cores machines.

In addition, I have also tried to increase heartbeat timeout intervals in
bamboo server <BAMBOO_HOME>/bin/setenv.sh as following "${JVM_SUPPORT_RECOMMENDED_ARGS:="bamboo.agent.heartbeatTimeoutSeconds=1200"}". And also added 'wrapper.successful_invocation_time=3, wrapper.max_failed_invocations=720
' to remote agent's wrapper.conf .

However, the change of heartbeat timeout settings on bamboo server and remote agents doesn't apply, and the timeout time is till 10mins(600s) by default values.

After several tries to catch the network stats, the TCP send Q both accumulated to around 6000 on both bamboo server and remote agent before previous condition happens. And 8085 http port is not reachable.

We are using bamboos to build 65 mins duration C++ builds with bamboo server + 5 remote agents.

Appreciated if anyone can give me some insight or help

 

0 answers

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bamboo

Bamboo Data Center is here!

G’day Bamboo customers, The wait is over! Bamboo Data Center edition you have been asking for is finally here! This self-managed enterprise offering of Bamboo has the resilience, reliability, and s...

558 views 1 17
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you