Stash slowdowns

Question

Hi,

we've been using Stash for a while (upgraded to multiple versions, we're now on 3.1.3)

We're having an issue where Stash will just slowdown, and the web interface responses really slowly and then just grinds to an halt. Running a TOP -C will show us that there is almost 100% memory usage. We can't find anything in the logs as well.

We have a license for 500 users but we have about 300 accounts created.

We are running Linux RedHat 6.4 with 2 VCPUs and 8 gigs of ram.

Any help would be appreciated,

Thanks!

Marc,

Marc-Andre · Answer

Hi Michael,

Thanks for helping out (thanks Mike as well)

Give me a few moments to answer all your questions, i'll get the info from the server right now.

Marc-Andre · Answer

Hi Mike, that's exactly the page that I'm reading right now. I haven't seen anything in the logs other than error message from the license that can't reach the marketplace website (We run the STASH machine in a envinronment without internet).

Michael Heemskerk · Answer

Hi Marc,

The type of slowdowns you're seeing and the fact that almost 100% memory is being used is an indication that the Stash process/memory is being swapped in and out of RAM.

In your top -C can you see which processes are using the memory? My guess is that it might be the git processes that are being executed when you clone a repository. The Stash process itself is designed to be able to run with a modest footprint, but the git processes that are forked by Stash are much more variable. Do you by any chance have large repositories that are cloned at (relatively) high frequency?

Because of the CPU and memory  usage of these git processes, Stash limits the amount of git processes that it will execute concurrently. See https://confluence.atlassian.com/display/STASH/Scaling+Stash for more background information. If you've changed the configuration option that controls the amount of concurrency (throttle.resource.scm-hosting), you'll either have to scale that value back or add more memory to the machine.

Other questions:

* How much memory have you allocated for the Stash process? 1GB should be enough.
* Is the server dedicated to Stash or do you have other applications running on it as well?
* When Stash is experiencing this type of slowdown, how may git processes do you have running? Please note, that each 'clone' operation typically results in a tree of 5 git processes (git itself forks child processes).
* Have you tweaked any of the settings in STASH_HOME/stash-config.properties? For instance, throttle.resource.scm-hosting?

Cheers,

Michael