Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Making Fisheye more responsive during high load - how to tweak performance of Jetty?

Hi

As a team we often observe low responsiveness of Fisheye , this looks like when it is hammered it often will not recover itself from that event. Only restart is the current solution to such problem in our case.

I have already read Atlassians article on how to deal with performance but this did not help much (we have beefy server ,lots of RAM etc).

Recently found this article as FeCru is said to use Jetty as application server.

https://wiki.eclipse.org/Jetty/Howto/High_Load

 

Our current TCP stack details ( 1Gb card)

x@y:~> sysctl net.core.rmem_max

net.core.rmem_max = 124928

x@y:~> sysctl net.core.wmem_max

net.core.wmem_max = 124928

x@y:~> sysctl net.core.somaxconn

net.core.somaxconn = 128

x@y:~> sysctl  net.ipv4.tcp_max_syn_backlog

net.ipv4.tcp_max_syn_backlog = 2048

x@y:~> sysctl net.core.netdev_max_backlog

net.core.netdev_max_backlog = 1000  

 

So having read the article I thought why not to tune Jetty (the one bundled).

Currently we have such config.xml :

...

<web-server context="/fisheye" site-url="xxxxxxxxxx.net/fisheye" min-threads="10" max-threads="200">
<http bind=":8060" proxy-port="443" proxy-scheme="https" proxy-host="xxxxxxxxxxxx.net"/>
<ajp13 bind=":8061"/>
</web-server>

...

Tried to follow the hint about limiting the default unlimited task queue (check Jetty wiki above) so put jenkins-web.xml and later jetty.xml or even jenkins-server.xml - like here:

 

[x ~]$ nano /cs/devtooling/packages/fecru/content/WEB-INF/jetty.xml (and others)

<Configure id="Server" class="org.eclipse.jetty.server.Server">
<Set name="ThreadPool">
<New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
<!-- specify a bounded queue -->
<Arg>
<New class="java.util.concurrent.ArrayBlockingQueue">
<Arg type="int">6000</Arg>
</New>
</Arg>
<Set name="minThreads">10</Set>
<Set name="maxThreads">200</Set>
<Set name="detailedDump">false</Set>
</New>
</Set>
</Configure>

 

 However I don't see any settings to be applied from this xml file, like those lines are simply ignored. 

Only having named the file jenkins-web.xml and pasting such xml makes Fisheye fail on start. Fisheye out in that case:

Exception in thread "Thread-1" org.springframework.context.ApplicationContextException: Application Context not initialised.
at com.cenqua.fisheye.config.SpringContext.getApplicationContext(SpringContext.java:51)
at com.cenqua.fisheye.config.SpringContext.getComponentByClass(SpringContext.java:90)
at com.cenqua.fisheye.config.RootConfig.getRepositoryManager(RootConfig.java:600)
at com.cenqua.fisheye.config.RootConfig.shutDownRepositories(RootConfig.java:1287)
at com.cenqua.fisheye.ctl.ShutdownService.stopImpl(ShutdownService.java:281)
at com.cenqua.fisheye.ctl.ShutdownService.access$000(ShutdownService.java:35)
at com.cenqua.fisheye.ctl.ShutdownService$1.run(ShutdownService.java:59)

 

Again we observe the behavior described here (that is a root cause that makes as sad) 

https://wiki.eclipse.org/Jetty/Howto/High_Load - last sentences basically , so does our Fisheye! It won't recover if to many users or automated requests simply hammer it. Config.xml settings work but even having 600 threads set we can reach this value and this means a death to the service. We dont observe much CPU or RAM consumed on the machine. We still have computing powers simply.

Any ideas ? Can we tweak Jetty ourselves with a bundled Jetty or this is not possible?

I could enable extra login via using jetty-web.xml but I failed on setting threads or task queue limit as it is suggested on the site linked.

 

TIA

1 answer

0 votes
Ben Stuart Atlassian Team Apr 24, 2018

Do you have a support license? I would recommend you open a ticket so that Atlassian can investigate your instance configuration and logs. These issues can be pretty complex and it may be difficult to troubleshoot in this type of forum. I can say that in my 3 years supporting FishEye/ Crucible, the solution to these types of cases have never been to modify Jetty. There are a few things that aren't clear from your description. 

  1. What is hanging when you have to restart FishEye/ Crucible? 
  2. What kinds of requests are you seeing hammering the app and what leads you to believe this? 

Are you seeing any errors similar to Indexer Paused with "Too many open files" Error? This might be a good thing to check. Also, have you checked your database connections? What kind of database are you using? 

We would typically scan your logs during the time of FishEye freezing for errors and compare that with thread dumps at the same time. If you have a support license please enable debug logging, wait for the incident to occur and then take 6 thread dumps at 5 second intervals and upload them, along with a support zip, to the ticket. 

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Confluence Cloud

🎨 Add some visual life to your templates

Hi Atlassian Community, My name is Avni Barman, and I am a Product Manager on the Confluence Cloud team. Based on feedback from you, we are giving admins more power to create templates that a...

247 views 5 9
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you