git.exe hangs builds when repository updated

[Ryan Schwoerer]

Following an update to the git repository the build for all branches of the repo get stuck 'checking out source code'.

There is a stuck 'git.exe' process on the server attempting to run `git.exe ls-remote' on the address '127.0.0.1:60460'.

Our git repo is running on Bitbucket Server on a different server, not on localhost.

If this 'git.exe' process is killed, the build picks up and runs as normal. I can see multiple other 'git.exe' command running successfully.

I cannot find where this particular instance of the command is getting the localhost address from. I've checked all the application links, and recreated them. As well as re-saving the linked repository, multiple times.

Bamboo version 6.9.2 build 60911 - 04 Jul 19, running on Windows 7.

--- [edit] ---

I also see system errors quite frequently from the same command.

[Daniel Santos]

Hi @Ryan Schwoerer

• git version - I have 2.22 installed already. Its a bit unclear which versions Bamboo supports. I know the last time I updated Bamboo it did not support the absolute latest git version.

Yes, we need to improve this in our documents. We have a feature request for this at [BAM-20511] Update supported git version documentation.
The document you should use to check this is Bamboo Supported platforms.

• triggers - All triggers are now Bitbucket triggered. The git process still hangs checking out source code during the build however.
• repo cache - just did this, and re-triggered another build via a git commit.

Thank you for covering those topics. At least we know the problem is not caused by them.

It looks like the repo triggered run is the only one that gets hung.

It is difficult to know what exactly is triggering the issue, but it is a good assumption.
Do you have multiple plans using the same repository?
Is this the only source repository where the issue happens?
Have you tried changing it to user and password, instead of using SSH, just for testing purposes?

I recently triggered another build, via a git push to Bitbucket.

This time the build hung again in 'checking out source code' (the step just prior to the actual build being created in Bamboo, meaning there is not a build number assigned yet)

I updated git to the latest from GitForWindows.org and a similar hang occured. on the git fetch call this time.

Thank you for the info. Another puzzle piece.
I had a conversation with other engineers yesterday about this problem and they asked if Bamboo is hosted on a Virtual machine or something that could be limiting the resources of the server regarding the number of git process that could be run.
Other questions:
   Are you using git bash on windows?
   When you tested this on the terminal, where you using SSH?

[Ryan Schwoerer]

Yes I have two plans on the same repo, however they each watch different branches.

I only have one source repo, so I can't comment on this being the only one or not.

The computer is Windows 7, Its a 6 core Xeon with 6GB of ram, and Bamboo is really the only thing running, I don't think it is working that hard.

I don't know about git bash, I followed the defaults installing git.

And I tested in the build server's terminal, so no SSH. I just ran the same command Bamboo is running.

Is there any logs or anything I can look at to see what Bamboo is doing?

[Daniel Santos]

Hi @Ryan Schwoerer

I agree with you that this is not a performance issue. At least from your description, it does not look like one. 

I'm pretty sure this is related to SSH handling on windows. You could try to avoid that as a test by using GIT repository type (not bitbucket server) and adding this repository with user/password credentials.

The logs you can check are in <Bamboo_Home>/logs/atlassian-bamboo.log. Please check if this file has any extra entries near the time of the old hangs. That could bring us some extra clues to move forward.

[Daniel Santos]

Hi @Ryan Schwoerer

There are a couple of things I would check to start narrowing down this one:

1. What is the difference between the plans working and this one?
   Is this a new configuration?
   Is this working on a specific remote agent?
   Is this one using Bitbucket server while the others use Cloud?
   Is this a repo configured a long time ago?
   
   
2. I'm not sure if resaving the repository will update the URL, assuming there was a change in Bitbucket server configuration after you added the repository in Bamboo.
   If you go to Bitbucket admin >> Server settings >> SSH access section, you should see how the SSH is configured and that configuration is the one used by a new repository in Bamboo.
   
   What you can try here is to create a Dummie plan and configure the repository again (a new linked or plan repository configuration pointing to ecs1.git) and check if the problem will still persist.
   
   
3. What are the log messages you see in the Agent logs when this problem happens?
   
   

I think the checks above will bring more clarity on what is causing this issue.

[Ryan Schwoerer]

Thanks for the help!
I answered your questions and added a bit more information.........

1. What is the difference between the plans working and this one?
- This plan had the Trigger set to 'Repository polling', I switched this back to 'Bitbucket server repository triggered'.
This seems to have resolved the issue where all the plan branches hang 'checking out source code', although the git process still hangs, see more below.

Is this a new configuration?
- This is not a new configuration. This has actually been an ongoing issue for quite some time, I only recently noticed the hung git process.

Is this working on a specific remote agent?
- I am not using remote agents.

Is this one using Bitbucket server while the others use Cloud?
- This repo is Bitbucket Server, we don't use Cloud.

Is this a repo configured a long time ago?
- Yes.

2. I'm not sure if resaving the repository will update the URL, assuming there was a change in Bitbucket server configuration after you added the repository in Bamboo.
If you go to Bitbucket admin >> Server settings >> SSH access section, you should see how the SSH is configured and that configuration is the one used by a new repository in Bamboo.

What you can try here is to create a Dummie plan and configure the repository again (a new linked or plan repository configuration pointing to ecs1.git) and check if the problem will still persist.

Changing the trigger seems to have changed the behavior.

3. What are the log messages you see in the Agent logs when this problem happens?
- I don't see any logs for the specific local agent, where are those located?
When the plan trigger was set to 'Repository polling' I was getting constant "Unable to detect changes" errors (shown above).

After changing the plan build trigger, the branches no longer hang 'checking out source code', however when the build starts it appears to have a hard time checking out from the server. It will hang on this line forever....

27-Aug-2019 09:44:43 Fetching 'refs/heads/*' from 'ssh://git@stp-sw.veeco.com:7999/ecs/ecs1.git'.
27-Aug-2019 09:44:43 Warning: Permanently added '[127.0.0.1]:60460' (RSA) to the list of known hosts.
27-Aug-2019 09:44:45 Checking out revision 6f6176487931b7aa4c869172a40752fe204a6b12.

I tried to look into the "Permanently added '[127.0.0.1]..." error, without much luck on what specifically to look for.

Thanks again for the help!

[Daniel Santos]

Hi Ryan,

Thank you for all those details (they are really important) and I'm sorry for the late response.

You should always prefer Bitbucket server trigger over the Polling trigger. The polling trigger will create more work for Bamboo and therefore more git processes running in parallel over time which could be one of the reasons this issue is happening.

The log lines you shared only show that Git is checking out the code when it hangs.

Actions I would take to try mitigating this:

• Update the git version installed in the Bamboo server machine.
  Newer versions of git usually bring stability and performance improvements.
• Convert polling triggers to Bitbucket server ones.
  As explained before this will reduce concurrency and improve performance.
  If you need to check what are the plans using polling trigger at the moment, please use some of the queries available in the table at the end of this document: How do I work with Triggering builds - Atlassian Documentation
• Reset the repository caches on affected plans.
  Bamboo keeps a repository cache for every source repository configured in Bamboo. If you want to make sure there is no problem with it, you can go to Admin >> Repository Settings >> Manage Caches >> (selected repo) Delete.

Let me know if those steps will fix the issues completely. 

[Ryan Schwoerer]

• git version - I have 2.22 installed already. Its a bit unclear which versions Bamboo supports. I know the last time I updated Bamboo it did not support the absolute latest git version.
• triggers - All triggers are now Bitbucket triggered. The git process still hangs checking out source code during the build however.
• repo cache - just did this, and re-triggered another build via a git commit.

After clearing the cache and retriggering a build, the new build is stuck on a similar line as above, it looks like the git fetch command hung.

manually running git fetch appears successful, and the repo is up to date.

I was about to ask if I could disable the cacheing, but a quick search tells me that was a feature request from years ago that was never added.

If I stop the build via Bamboo, and rerun it via Bamboo it runs just fine.

It looks like the repo triggered run is the only one that gets hung.

[Ryan Schwoerer]

I recently triggered another build, via a git push to Bitbucket.

This time the build hung again in 'checking out source code' (the step just prior to the actual build being created in Bamboo, meaning there is not a build number assigned yet)

I updated git to the latest from GitForWindows.org and a similar hang occured. on the git fetch call this time.