Long build queue time

[Marc Zych]

I've been experiencing longer and longer build queue times recently. It happens on all plans, but one plan in particular is problemsome because it is a single-file merge queue that we use as a gatekeeper for merging pulls into master. We manually queue custom builds using the API and they routinely sit there with a message like so:

> Build:#301 has been queued for 11 minutes

This is supremely frustrating when there are a few pulls waiting to be tested and merged but they are mysteriously left sitting in a queue waiting for seemingly nothing.

I'm seeing things like this in the Bamboo agent's logs:

INFO   | jvm 9    | 2016/11/09 18:05:08 | 2016-11-09 18:05:08,512 INFO [0-BAM::smc-2-cpu-01d.corp.<redacted>.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 67666427 checking build queue for executables...
INFO   | jvm 9    | 2016/11/09 18:10:08 | 2016-11-09 18:10:08,645 INFO [0-BAM::smc-2-cpu-01d.corp.<redacted>.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 67666427 checking build queue for executables...
INFO   | jvm 9    | 2016/11/09 18:13:24 | 2016-11-09 18:13:24,346 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] ExecutableQueueUpdate: addToQueue, agents known to be affected: []
INFO   | jvm 9    | 2016/11/09 18:13:24 | 2016-11-09 18:13:24,347 INFO [0-BAM::smc-2-cpu-01d.corp.<redacted>.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 67666427 checking build queue for executables...
INFO   | jvm 9    | 2016/11/09 18:13:31 | 2016-11-09 18:13:31,083 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] ExecutableQueueUpdate: addToQueue, agents known to be affected: []
INFO   | jvm 9    | 2016/11/09 18:13:31 | 2016-11-09 18:13:31,084 INFO [0-BAM::smc-2-cpu-01d.corp.<redacted>.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 67666427 checking build queue for executables...
INFO   | jvm 9    | 2016/11/09 18:16:36 | 2016-11-09 18:16:36,919 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] ExecutableQueueUpdate: addToQueue, agents known to be affected: []
INFO   | jvm 9    | 2016/11/09 18:16:36 | 2016-11-09 18:16:36,919 INFO [0-BAM::smc-2-cpu-01d.corp.<redacted>.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 67666427 checking build queue for executables...
INFO   | jvm 9    | 2016/11/09 18:21:37 | 2016-11-09 18:21:37,130 INFO [0-BAM::smc-2-cpu-01d.corp.<redacted>.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 67666427 checking build queue for executables...

Yet it doesn't pick up the job for quite some time. The current build finally hit 15 minutes in the queue and then the agent decided to consume it and it finished in under a minute. This is beginning to be a serious hinderance in our ability to develop code.

Any explanation would be greatly appreciated.

[Marc Zych]

I switched to Bamboo Server and there haven't been any delays between queueing and running on the remote agent so far.

[Marc Zych]

I'm using Bamboo Cloud so I don't have access to the server clock. However, the clock on the dedicated agent matches what is shown on www.time.gov.

[Sergey Podobry _Stellarity Software_]

I'm not sure but check whether the clocks are synchronized on the server and the agent.

[Marc Zych]

Anybody have an explanation? I'd love to fix this because the current behavior is a bit ridiculous.