Hello
We have a bitbucket server/data center (7.21.x, tested also 8.9.1 same) running and a build-system which massively retrieves data from this bitbucket server. The buildserver has several build-queues which run in parallel and cause peaks of about 150 git clone semi-parallel of about 800 consecutive downloads in a short amount of time.
I say semi-parallel, because the git clone commands are quickly coming to an end, because the download data is small, so that the buildserver launches a bunch of them, but they never really are running totally parallelized on bitbucket side. However the bitbucket server gets a lot of requests coming in and here is the problem.
At some point the bitbucket SSH server does not accept() new connections and so does not SYN/ACK the TCP connection. On the buildserver side we get a "connection timed out", which breaks our automated build process. The buildserver does consecutive retried, but once queue is full it is full and eventually the build process will fail due to this timeout.
The error message is *not* that the limit of parallel connections bitbucket accepts is reached! We tested that also, by generating a 130 git clone of large data volume in parallel. This is not a problem. The limits as configured with scaling.concurrency and throttle.resource.* are fine. If the limit is reached we get the error message that the limit is reached, not a "connection timed out"!
The problem seems to be that bitbucket does not like coming in such a huge amount of short git commands in a short period of time.
We boiled down the issue to the TCP SYN queue, which for port 7999 is limited to 50!
And we get a lot of "SYN to LISTEN sockets dropped" and "times the listen queue of a socket overflowed" (netstat -s)
So, is there a way to increase this limit to something higher, let's say 150?
To be clear, it is the value for backlog set by listen(fd, backlog)
I see no way how to modify this value. Is there maybe a way?
Thank you in advance
Peter
