Since upgrading from Bamboo 5 to Bamboo 6, we have seen several issues with build and deployment triggers. Sometimes hey hang, or they don't fire at all. Our plans and configurations increased too, so I'm not sure what the actual cause.
I've listed my Bamboo system configuration below:
Operating system: Windows Server 2012 R2 6.3
Operating system architecture: amd64
Available processors: 8
Java version: 1.8.0_172
Total memory: 1008 MB (--> this seems off, the server has 16GB of RAM)
Build number: 60509
Build date: 4/20/18
git version: 2.17.0.windows.1
I cannot give exact numbers, but we have around 80 projects, each containing one to 5 build plans. Each build plan has at least one repository branch, but plan branches for at least half of them can go as high as 5 at the same time. Let's say 75% of all projects have a deployment plan too, each having 3 environments (DEV, QA, PROD). Every build plan is using git polling (default 180 seconds timeout) and each deployment plan for DEV and QA has a trigger so that it fires after a successful build of the corresponding plan branch.
Now, the last couple of months we have observed the following:
We are running a single Bamboo instance, with 3 local agents and 1 remote agent. The remote agent is not configured with build capabilities, and can thus only be used for some deployment plans.
I searched the bug tracker, but I could not find any similar issue.
Is it possible we have too much plans and configurations for a single instance? Our the server resources insufficient? Is there additional information I could provide to help track down this issue?
From what you've described; the hanging / long running Git processes seem to be the most likely culprit here.
To explain, Bamboo has a couple of thread pools for actions like change detection, branch detection and general plan execution steps.
When we see huge delays in triggering, usually it's because these thread pools are tied up waiting for something. Long running / hung Git processes are the main offenders.
My initial pointers for now:
It can be a bit tricky to troubleshoot and we may need to open up a support ticket to help out further but take a look into some of those things and let us know how it goes. :)
Thanks for the quick reply.
Unfortunately, none of the changes to my Bamboo installation seem to have had an impact. We still observe issues with with builds and deployments that are not triggered. I see no abnormal memory or CPU usage, but I do see a lot of "Git for Windows" processes active on the server. Both git.exe and git-https-remote.exe. Numbers can go as high as 50!
If I kill most of the processes, then sometimes Bamboo comes through and picks up a queued build, manual or automatic trigger doesn't make a difference. Sometimes I have to reboot the server before any new builds are picked up.
Is there any additional information I can provide?
G’day Community! As we gear up to introduce Bamboo Data Center to the world, we wanted to make sure that we shared a bit more about Bamboo, the product. Our team has put together an overview video ...
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events