A lot of git for-each-ref

[Alexey_Efimov]

Why our stash have follow every minute (it fork git for-each-ref and kill them)?

16261 stash 20 0 23432 9812 1028 R 24 0.0 0:00.71 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16266 stash 20 0 20784 7112 976 R 22 0.0 0:00.65 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16277 stash 20 0 20232 6580 976 R 20 0.0 0:00.61 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16276 stash 20 0 19836 6056 976 R 19 0.0 0:00.58 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16272 stash 20 0 19572 5792 976 R 18 0.0 0:00.55 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16284 stash 20 0 19572 5776 976 R 17 0.0 0:00.52 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16263 stash 20 0 19572 5788 976 R 17 0.0 0:00.50 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16271 stash 20 0 19572 5792 976 R 16 0.0 0:00.49 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16267 stash 20 0 19572 5792 976 R 16 0.0 0:00.47 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16269 stash 20 0 19572 5792 976 R 15 0.0 0:00.46 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16274 stash 20 0 19572 5788 976 R 15 0.0 0:00.45 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16268 stash 20 0 19572 5788 976 R 13 0.0 0:00.40 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02
16270 stash 20 0 19572 5788 976 R 13 0.0 0:00.40 /usr/bin/git for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02

[Ben Humphreys · Accepted solution]

Hi Alexey,

What were the circumstances under which you saw this activity? Was it just after a restart? Or perhaps a cluster node re-joining the cluster?

Were all the processes you show running concurrently? Or were only a few running at a time?

There are a few possible reasons, but the most likely is tag re-indexing. Bitbucket Server keeps a cache of tags, so it can annotate commits in the user interface with and associated tags.

It does this in a fairly low impact way; it processes your repositories serially so only a single "git for-each-ref" will be be running at a given time. Also this command is quite lightweight and on a typical repository will complete in a few 100 milliseconds at most. However for instances with large numbers of repositories tag re-indexing could easily run tens of minutes before all repositories are re-indexed.

I hope this helps explain what you are seeing. Certainly if this is causing problems with your instance please raise a support request at https://support.atlassian.com

Regards,
Ben

[Alexey_Efimov]

It fork and kill that processes permanently all time. It not really load CPU, but it interest why Stash run them again and again. And i never seen that Stash stop running that processes.

[Ben Humphreys]

Hi Alexey, There are a few different ways you can try and see what activities are causing these commands to be run, but perhaps the easiest is to enable profiling and then look for instances of "for-each-ref" in the atlassian-bitbucket-profiler.log (or atlassian-stash-profiler.log) log file.

[Ben Humphreys]

Also, I was going to ask.... how do you know Stash is killing the processes? If the process runs for more than 60s (or perhaps 120s?) it will be killed and an error will be logged in atlassian-bitbucket.log: "An error occurred while executing an external process: process timed out" Since that particular command should complete very quickly, it would be abnormal to see this for instances of "git for-each-ref"

[Alexey_Efimov]

> how do you know Stash is killing the processes? strace to some of them show that process was terminated by SIGTERM. The processes is running really short time, less that minute.

[Alexey_Efimov]

The root of this problem% we have plugin that continue check each opened pull request. It have method to find user who have permission to merge that pull request. Inside of this method it has invocation to com.atlassian.stash.repository.ref.restriction.RefRestrictionService#canWrite. Exactly this method generated dozen of checks and git for-each-ref. And it work more that minutes for each invocation.