I am trying to set up a Stash Datacenter after the singular Stash install almost bombed last week.
To determine my specs of my new setup ,I wish to test it first. Is there a plugin/script for this purpose?
We don't have load or performance testing scripts available at the moment that work outside of the specific test lab that we used here. Also, we find that each customer's usage differs quite a bit, so no one test setup applies well to all situations. That said, there are a few options available...
Hope that helps!
The scripts that were used for testing Stash DataCenter, were developed for use with a proprietary stress testing and orchestration framework. The scripts were developed in Scala and ran on our framework, which makes it very difficult to publish them for public consumption.
I created a project that can be used as a base for load scripts, to assist people, who want to replicate our results. I am open to receiving pull requests to make the scripts better
The scripts are here: https://bitbucket.org/Dostanian/gatling-4-stash
I hope this provides a good starting point for getting your own scripts up and running.
The omission of units was deliberate, because every team has their own usage pattern. The shown results are representative of how Atlassian and other large customers (determined by collaboration) use Stash.
The mix of web and Git traffic was roughly made up of the following:
To generate enough load we used a cluster of loader machines spawning between 200 and 800 threads. Each thread would execute one of the operations defined above and once complete be scheduled to execute another as soon as possible. We used Apache HTTP client instead of spawning web browsers.
It is important to reduce the number of authentication requests to Stash when stress testing. Authenticating on each request degrades performance dramatically and might exhaust your database connection pool. We used a cookie manager to ensure that a thread was authenticated once before doing a batch of work.
The network usage hit around 300 MB/s sustained between loaders and cluster nodes when cloning. This figure was as a result of the throughput of the NFS server used and not the network bandwidth.
Stash DC performance is directly reliant on how well your NFS server copes with servicing requests from Stash.
The saturation point is determined by the number of connection handlers in Apache Tomcat and number of connections in the database connection pool. Additionally there is a hard limit on the number of Git hosting and command tickets. When all Git tickets are exhausted, wait times will increase and in the worst case time out.
When all limits were increased beyond safe limits the first resource to suffer saturation was CPU, then Disk, Network and finally Memory. With little to no protection the system would eventually be saturated and enter a period of extremely slow response times. It would recover in approximately 30 seconds to a minute or two later and then resume normal operation. This was related to the OS having to time out sockets and Stash recycling database connections.
With limits in place the server would remain operational during excessive load periods, but as a side effect time outs would be encountered. Memory was largely used for disk cache. The most aggressive consumer of Memory and CPU is Git executable.
I hope this answers your question,
Hi Vish, could you elaborate on what you are trying to achieve with the tests? What metrics are you looking for? Are you looking for scripts to help you set up Stash Data Center? Load-test Stash Data Center on your new setup? The more details you provide, the more accurate of an answer people will be able to give you :)
hi Felix - thanks for the response. here is what my ask is: - We are in the process of setting up a Stash Datacenter which shall be replacing the existing single node installation of stash. - I wish to have some realistic capacity/resource planning for current setup, as well as projections for future growth. As a starting point we wanted to load test the current set up with varied server specs to determine the capacity now, which will give us some base line to plan projections for required resources. The aim is to have an optimal design for Stash data center. thx, Vish
Felix, Roger, Charles:
Thanks for your prompt response.
Charles: I'll try and socialize Gatling with my team here.
Until then, I realized I might have to rephrase my query/request:
If such info is available and can that be shared, it'll help put things in perspective.
Apologies for the delay in responding to this thread. The links you provided on 27th July was very informative.
Qs I have now are:
Hello! My name is Mark Askew and I am a Premier Support Engineer for products Bitbucket Server/Data Center, Fisheye & Crucible. Today, I want to bring the discussion that Jennifer, Matt, and ...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs