Is there a plugin to perform a load test/performance test on Stash?

[Vish Kul]

I am trying to set up a Stash Datacenter after the singular Stash install almost bombed last week.

To determine my specs of my new setup ,I wish to test it first. Is there a plugin/script for this purpose?

thx,

Vish

[Charles Olivier]

Hi Vish,

The omission of units was deliberate, because every team has their own usage pattern. The shown results are representative of how Atlassian and other large customers (determined by collaboration) use Stash.

The mix of web and Git traffic was roughly made up of the following:

• 20% CI Git polling (check for changes)
• 20% CI Git shallow clone
• 15% Git clone
• 15% Pull Request web page activity
• 15% Commonly used web pages
• 15% Git fetch

To generate enough load we used a cluster of loader machines spawning between 200 and 800 threads. Each thread would execute one of the operations defined above and once complete be scheduled to execute another as soon as possible. We used Apache HTTP client instead of spawning web browsers.

It is important to reduce the number of authentication requests to Stash when stress testing. Authenticating on each request degrades performance dramatically and might exhaust your database connection pool. We used a cookie manager to ensure that a thread was authenticated once before doing a batch of work.

The network usage hit around 300 MB/s sustained between loaders and cluster nodes when cloning. This figure was as a result of the throughput of the NFS server used and not the network bandwidth.

Stash DC performance is directly reliant on how well your NFS server copes with servicing requests from Stash.

The saturation point is determined by the number of connection handlers in Apache Tomcat and number of connections in the database connection pool. Additionally there is a hard limit on the number of Git hosting and command tickets. When all Git tickets are exhausted, wait times will increase and in the worst case time out.

When all limits were increased beyond safe limits the first resource to suffer saturation was CPU, then Disk, Network and finally Memory. With little to no protection the system would eventually be saturated and enter a period of extremely slow response times. It would recover in approximately 30 seconds to a minute or two later and then resume normal operation. This was related to the OS having to time out sockets and Stash recycling database connections.

With limits in place the server would remain operational during excessive load periods, but as a side effect time outs would be encountered. Memory was largely used for disk cache. The most aggressive consumer of Memory and CPU is Git executable.

A more detailed explanation of throttling is given here: Stash Resource Throttling and here Stash is reaching resource limits.

I hope this answers your question,
Charles Olivier

[Charles Olivier]

Hi,

The scripts that were used for testing Stash DataCenter, were developed for use with a proprietary stress testing and orchestration framework. The scripts were developed in Scala and ran on our framework, which makes it very difficult to publish them for public consumption.

I created a project that can be used as a base for load scripts, to assist people, who want to replicate our results. I am open to receiving pull requests to make the scripts better 

The scripts are here: https://bitbucket.org/Dostanian/gatling-4-stash

I hope this provides a good starting point for getting your own scripts up and running.

[Roger Barnes]

Hi Vish,

We don't have load or performance testing scripts available at the moment that work outside of the specific test lab that we used here. Also, we find that each customer's usage differs quite a bit, so no one test setup applies well to all situations. That said, there are a few options available...

• We know of several customers that run simulated load testing of common operations based on their current usage. Our Technical Account Management service can provide guidance for such initiatives
• Our expert partners can offer personalised services, including performance/load testing
• Atlassian's support team can also help with diagnosing and advising on performance issues. This might be a good place to start with understanding your current and forecasted usage.

Hope that helps!

[Vish Kul]

Apologies for the delay in responding to this thread. The links you provided on 27th July was very informative.

Qs I have now are:

• There is a graph of Throughput Vs Number of concurrent browsers. While we can see the trend of the tests, there are no units specified for the X and Y axis.
• At the threshold point, which of the system resource seems to be max'd out? It is CPU or RAM or something else ?

thanks,

Vish

 TestResults.png

[Vish Kul]

Felix, Roger, Charles:

Thanks for your prompt response.

Charles: I'll try and socialize Gatling with my team here.

Until then, I realized I might have to rephrase my query/request:

1. Is there any base-line or benchmark performance data on your Test Data Center environment with the details of HW/SW specs?
2. During the performance test phase for Stash, what were the bottlenecks faced and what was done to tune ?

If such info is available and can that be shared, it'll help put things in perspective.

regards,

Vish

[Vish Kul]

hi Felix - thanks for the response. here is what my ask is: - We are in the process of setting up a Stash Datacenter which shall be replacing the existing single node installation of stash. - I wish to have some realistic capacity/resource planning for current setup, as well as projections for future growth. As a starting point we wanted to load test the current set up with varied server specs to determine the capacity now, which will give us some base line to plan projections for required resources. The aim is to have an optimal design for Stash data center. thx, Vish

[Felix]

Hi Vish, could you elaborate on what you are trying to achieve with the tests? What metrics are you looking for? Are you looking for scripts to help you set up Stash Data Center? Load-test Stash Data Center on your new setup? The more details you provide, the more accurate of an answer people will be able to give you :)