Houston, we have a problem! What an APM tool we will use to find a bottlenecks?

[#1]

Hi! 

I would like to share a series of articles related to the APM( Application performance monitoring) tool. It based on my investigation of different problems related to the performance. 

Let's start from most popular in industry tool "New Relic". 

1. We have a business case: it is the angry end of users :) 

2. Requirements:

2.1. ASAP detect the bottleneck

2.2. Continuously monitoring the problem

2.3. Possibility to review (on browser side, on backend side and RDMS side as well)

2.4. Price and Functionality

Let's start from scratch. 

1. We need to register and install ab agent in our situation is java. 

You will see below panel.

I have spent on test environment around 5 min to install. 

2. For the realistic situation, I have installed an app with the leak. Where immediately NewRelic detects the problem:

 3. On the tab of the transaction, you can find the slowest average response time: 

4. After we that we can investigate not web type: 

5.  And next good things are based on the external services (it helps to be easy to understand the problem with integrations). And next awesome things is Error analytics: 

6. Sometimes you can meet with the situation where server metrics are health, and end the user is feeling the problem. Therefore I totally recommend you to try the browser analytic for check full situation. Where you can easily understand problem-related with GEO, version of browser or specific app with JS:

7. You can find many and many metrics from scratch:

e.g. that helped to me undestand the problem of js scripts related to the UI customisations

As resume: 

1. I have found many bottlenecks on backend and frontend side by that app. 

2. I have worked with the next apps

• APM - https://newrelic.com/products/application-monitoring/  https://docs.newrelic.com/docs/apm/new-relic-apm/getting-started/introduction-new-relic-apm

• BROWSER - https://newrelic.com/products/browser-monitoring/  https://docs.newrelic.com/docs/browser/new-relic-browser/getting-started/introduction-new-relic-browser

3. Price is high, but you can speak.

4. I am going to test the next tool. Feel free add here your favorite APM tool, hence I will try to check and review that one for my Atlassian infra :)

I hope it was interesting to you.

Thanks for you attention.

If it will be interesting I am planning to review Dynatrace

Cheers,

Gonchik Tsymzhitov

original post

[Mauricio GALVEZ]

Great article

but in my humble opinion I found DynaTrace most powerful rather than NewRelic.. btw NewRelic is a great product too.

[Jens Niedrich]

Hi Gonchik, 

we were using NewRelic in a client situation and were forced to switch over to Instana as the leading tool. I would like to know your feedback on this tool since the client and a lot of other stakeholders consider Instana is the better APM. I personally can say that NewRelic is providing better analysis on database metrics but can not compete in JVM monitoring.

[Fabian Warthenpfuhl]

Cool article, it's always great seeing people bring attention to APM topics.

In case you want to check out open source APM technologies as well, have a look at https://openapm.io/

Our very own open source Java agent (https://www.inspectit.rocks/) should work with the popular Atlassian tools as well.

Regarding the commercial options, there is a strong wealth of vendors and tools to choose from, proven enterprise products as well as as new, innovative players. In our experience, they each have their strengths (and shortcomings) and finding the best solution heavily depends on your individual requirements and use cases.

[Jack [AppFox]]

Thanks Gonchik for sharing a real-world example of NewRelic in respect to Atlassian apps.

I haven't seen the RUM/AUM features of NewRelic before, but based on your screenshots, it looks incredibly useful for finding regional bottlenecks - the graphs for the RUM especially.

This in combination with the new CDN features coming to the Data Center applications (currently only Confluence) will make a powerful tool in reducing latency in global companies.

Another commercial solution I would recommend adding to any list of comparison is AppDynamics - which provides the same metrics including JMX, JVM, RUM/AUM and Health Checks, but the price is more prohibitive.

I have included some screenshots below to help compare the tools. (re: the stats - this is not a real world example) - would also love to see some examples of using DynaTrace if anyone has any?

1. A Dashboard for an entire Atlassian Stack:

2. Jira Transactions by Load, Errors, Response Time etc:

3. Service Map of all Atlassian Apps and DB:

4. Real User Monitoring for JS etc.

[Gonchik Tsymzhitov]

Hi! 

Thanks for all sharing different tools. I am going to try to use a different tools. 

Nowadays, I would prefer a tool which will help me to fix easier the exist pain. Fortunalte, graphs, metrics are a lot of in real world.

[Jamie Rogers]

Hi @Gonchik Tsymzhitov 

We recently moved to Dynatrace SaaS solution for our entire stack and this has proven to us faster root cause using their problem analysis A.I engine.

I am looking forward to your review about Dynatrace.

[David Yu]

@Jens NiedrichInstana looks very interesting. Thanks for the tip.

[Mikhail Zalygalin]

@Jack [AppFox] , сould you please share your expirience with AppDyn?
We added 

CATALINA_OPTS="${CATALINA_OPTS} -javaagent:/opt/appagent/javaagent.jar"

to the setenv.sh file.  

CATALINA_OPTS="${GC_JVM_PARAMETERS} ${CATALINA_OPTS}"
CATALINA_OPTS="${CATALINA_OPTS} -javaagent:/opt/appagent/javaagent.jar"
export CATALINA_OPTS

And with this option JIRA is unable to start.

[Mikhail Zalygalin]

@Jack [AppFox]  could you please share your expirience with AppDyn?

I used the following config in the setenv.sh

CATALINA_OPTS="${GC_JVM_PARAMETERS} ${CATALINA_OPTS}"
CATALINA_OPTS="${CATALINA_OPTS} -javaagent:/opt/appagent/javaagent.jar"
export CATALINA_OPTS

But Jira is unable to start. All permissions granted.

[Jack [AppFox]]

Hi @Mikhail Zalygalin

If you do not add the 'bootdelegation' environmental variables, then the application will fail to start as the AppDynamics agent will cause issues.

This is discussed on the following pages:

https://confluence.atlassian.com/jirakb/using-javaagent-with-jira-790793295.html

https://confluence.atlassian.com/confkb/confluence-will-not-start-after-adding-appdynamics-to-instance-858768500.html

Hopefully that helps you get it running!

All the best,

Jack

[Mikhail Zalygalin]

Thanks a lot @Jack [AppFox] !

It works.

[Rob Jahn]

@Gonchik Tsymzhitov  -- Feel free to reach out to me for any support or questions in your Dynatrace evaluation.

[Gonchik Tsymzhitov]

@Rob Jahn  I am happy to reinvestigate again, I am going to type you a direct message