Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

heartbeat/broker timeout, communication issue between server and agents

We currently experiencing a problem on my Bamboo Server 6.9 for heart beat issues show on following remote agent logs, and after it reaches 10 mins timeout, the bamboo server just show 'Agent 'remote-bamboo-agent-01' went offline while building xxxxx'. The results of that build will not be available.'

INFO | jvm 5 | 2020/12/26 02:14:29 | 2020-12-26 02:14:29,652 INFO [scheduler_Worker-4] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 3 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:15:29 | 2020-12-26 02:15:29,651 INFO [scheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 4 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:16:29 | 2020-12-26 02:16:29,652 INFO [scheduler_Worker-9] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 5 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:17:29 | 2020-12-26 02:17:29,652 INFO [scheduler_Worker-2] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 6 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:18:29 | 2020-12-26 02:18:29,653 INFO [scheduler_Worker-3] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 7 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:19:29 | 2020-12-26 02:19:29,652 INFO [scheduler_Worker-5] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 8 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:20:29 | 2020-12-26 02:20:29,652 INFO [scheduler_Worker-7] [AgentHeartBeatJob
..............................
INFO | jvm 5 | 2020/12/26 02:23:29 | 2020-12-26 02:23:29,660 FATAL [scheduler_Worker-10] [AgentContext] Agent encountered a fatal error: 'The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent'. Exiting.
INFO | jvm 5 | 2020/12/26 02:23:29 | java.util.concurrent.TimeoutException: The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent

I have tried the following measures as mentioned on bamboo server documentations:
1. Increase bamboo server heap size to 16GB
2. Delete all build logs in <BAMBOO_HOME>/xml-data/builds
3. Sync bamboo servers and remote agent to same NTP servers

All remote agents are solo instance on 16 cores 32GB machines, and the Atlassian suite is also hosted on an 64 cores machines.

In addition, I have also tried to increase heartbeat timeout intervals in
bamboo server <BAMBOO_HOME>/bin/setenv.sh as following "${JVM_SUPPORT_RECOMMENDED_ARGS:="bamboo.agent.heartbeatTimeoutSeconds=1200"}". And also added 'wrapper.successful_invocation_time=3, wrapper.max_failed_invocations=720
' to remote agent's wrapper.conf .

However, the change of heartbeat timeout settings on bamboo server and remote agents doesn't apply, and the timeout time is till 10mins(600s) by default values.

After several tries to catch the network stats, the TCP send Q both accumulated to around 6000 on both bamboo server and remote agent before previous condition happens. And 8085 http port is not reachable.

We are using bamboos to build 65 mins duration C++ builds with bamboo server + 5 remote agents.

Appreciated if anyone can give me some insight or help

 

0 answers

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bamboo

Bamboo 7.2 is here!

I'm happy to announce that Bamboo 7.2 has been released and it’s overflowing with awesome new features. This will be the last major Server release before the launch of Bamboo DC. Bamboo logs We...

142 views 2 5
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you