multiple tag push to repositories on bitbucket server - slow response

[Kevin Jones]

We're running Bitbucket server for a project that comprises about 20 repositories. Right now, we're using python scripts to iterate commands that need to be performed over all of the repositories (e.g. a developer needs to clone all of them).

When it comes to using this python script to do a git push origin --tags command, we'll get through a good chunk of the repositories, but then the script will just hang. Looking in bitbucket, we can see that the tags were committed to the repository, but there wasn't a response coming back to our script to tell it to move on.

We've taken steps to ensure that it's not an authentication issue (increasing the timeout of cached credentials), but this issue still seems to occur.

Pushing something like tags isn't that resource intensive, is it? Or is there a performance upgrade on the server (more memory) that I could do to assist with this?

Thanks,

Kevin

[Johannes Kilian · Accepted solution]

Just as a follow-up of the behaviour I described in my comment above - and the solution we found.

As I wrote above, pushing the tag was really fast (as the tag was available in the remote repository very soon after starting the push operation) but after the tag was pushed, the process was running on "forever". Looking into more details I saw, that in the remote repository onour Bitbucket server there are some repository hooks -  one of them called "postreceive-hook": This hook runs after the data is received (after receiving the pushed data, this hook triggers) This hook is used to trigger some finalizing server actions and is used by some addons to do their tasks ....

In our case we found the culprit with an AddOn we used: Smart Commit. SmartCommit is triggered by the postreceive-hook and tries to do its work ... but in case of a few single repositories it runs forever, making the push operation lasting very long (until timeout is reached).

Removing this addon everything works fine: git push runs without any timeout problems anymore ...

So the solution might be, disabling your addons  step by step to isolate the culprit ...

[Jeremy Carlson]

Disabling Smart Commit resolved this for us. Thanks for posting your research!

[Kevin Jones]

There isn't a specific hook that we found. What we did instead was to factor in a "sleep" counter when we tag a build. So we git push tags && sleep 0 to allow the system time to clear.

[Johannes Kilian]

@Jeremy Carlson: Finding the problem needed about 6 weeks in a production environment with help of Atlassian Support. As it was on our production server, it was problematic to find timeslots to do some tests ...

[Jeremy Carlson]

We are experiencing similar behavior. Whenever we push a tag, the tag gets to Bitbucket right away, but then git just sits there until a network timeout occurs. Pushing and pulling is fast, although some users report slowness when pushing a new branch. This behavior has existed since we first switched to Bitbucket Server eight months ago.

OS: Debian 8
Memory: 16G (with plenty free)
Server git: 2.1.4
Bitbucket Server: 4.8.1 

[Johannes Kilian]

Not a real answer, but just a confirmation that we do have the same problem here. On a quite large repository pushing a tag runs in timeout on the server. Looking in the pushed repo on the server we can see that the tag is alreasy available in the remote repo really fast - but the push process runs on for more time and ends up in a timeout ...