Monitoring of Atlassian applications with Prometheus

[#1]

Hi all,

In this article we will talk about real-time monitoring of Atlassian applications.

But first, let us define what monitoring is and why it is important to monitor Atlassian applications.

Monitoring is used for measuring performance and availability of software applications. Another words, it visualizes the current state of an application in the real-time mode.

When is it useful to monitor Atlassian applications?

Here are a couple of examples:

• You want to know how an Atlassian application update or a plugin update influences performance.
• You want to know hardware or memory parameters during certain conditions or certain time intervals. For example, how the number of users or day time influences performance.
• You want to monitor how active your users are. For example, how many issues are created within a day or an hour.
• You want to be notified if the license for an Atlassian application or a plugin is about to expire.
• You want to know how much disk space is taken by attachments.

 It is also important not only to be able to view information about the current state of an application, but also receive notifications if something went wrong.

Solution architecture

Prometheus can monitor thousands of services. Prometheus collects metrics as frequent as you defined, usting the pull-model, and stores collected metrics in the database. Also there is the Alert manager, which can send notifications to different channels, if an event has been triggered.

Prometheus exporter is an agent, which collects metrics from entities (a whole server or an application). Prometheus has rich capabilities for extending, that is why there are exporters for almost all popular applications.

[ jira | confluence | bitbucket | bamboo ]_exporter works the following way:

• collects information from the database of an Atlassian application;
• collects information from Java API;
• intercepts HTTP requests and collects information based on the requests;
• collects JMX metrics.

You can find an example of Prometheus/Grafana installation in Docker here. This example was kindly provided by Stefan Prodan.

Exporter installation and setup

First, you need to install the plugin for your Atlassian Application. You can find all plugins here. When the plugin has been installed, you can receive metrics by baseUrl/plugins/servlet/prometheus/metrics url.

You also can set up security keys.

Then you should add the reference to the metric collection servlet to the prometheus.yml. For example, for Jira it would look like this:

- job_name: 'jira'
  scheme: https # change to http if don't you have https
  metrics_path: '/jira/plugins/servlet/prometheus/metrics'
  static_configs:
  - targets: ['myhost:2990'] # Jira host and port you serve

 Installation and setup is finished. You can collect metrics.

Metrics

You can find the whole list of metrics in the documentation for the plugins: Jira, Confluence, Bitbucket, Bamboo.

Query examples:

• Uptime:
  
  

  jvm_uptime_gauge{job="jira"}

• Server response time:
  
  

  (sum(rate(jira_request_duration_on_path_sum[1m])) by (path)) / (sum(rate(jira_request_duration_on_path_count[1m])) by (path))

• Total number of issues:
  
  

  jira_total_issues_gauge

• Dashboard view number:
  
  

  sum(increase(jira_dashboard_view_count[1h]))

More information about queries you can find here.

Grafana dashboards

You can use Grafana to visualize metrics stored in Prometheus. There are a couple of example dashboards in the official site.

You can find a dashboard for Jira here. 

Here is a dashboard for Confluence:

Features

There are plans to let users define their own metircs, using the Groovy language. You will be able to collect your custom metrics not only from the Atlassian applications but also from other plugins. Also there will be metrics collected from Jira Service Desk.

All Prometheus exporters for Atlassian applications are open source and continuously gaining new features.

If you have ideas what features should be included in the plugins, kindly create tickets (Jira, Confluence, Bitbucket, Bamboo) or open pull requests.

Conclusion

If your Atlassian instances are large and you need to ensure their stability, then you need to use monitoring. It is up to you to decide how to do it. Prometheus is one of the possible solutions to accomplish this task.

Thank you!

[Charly _DEISER_]

Great post! thanks a lot for sharing this info. I have implemented it in our Jira instance and it works very well. One question, do you know if it's possible to show in the Requests duration gadget the info from all of our apps? e.g. eazyBI, Structure, etc.

Regards

[Alexey Matveev]

Thank you for the comment! The guy who develops the plugin said that he will add the ability to get metrics from other plugins. 

[Charly _DEISER_]

Great!!!! thanks.

[seniorandreymarkelov]

@Charly _DEISER_ Hi Carlos. Currently I expose only 1st level of paths: rest, servlet, projects, etc. But the next release will also custom paths with n levels. It will fully cover your needs.

[Charly _DEISER_]

Thanks @seniorandreymarkelov That will be really useful!

regards 

[Daniel Holmes]

Interesting.  We are building a similar solution based on ELK stack based on presentation we saw at Summit last fall.  Even though we aren't done with the visualization part, just being able to have all out logs routed and queryable in Elastic search has already help us track down a couple issues.

[seniorandreymarkelov]

@Daniel Holmes Daniel, sure it will work but it doesnt allow combine with system metrics like swapping etc. Also prometheus has alert manager. Graylog has it but also not fully specialised for that.

[Daniel Holmes]

Elastic has a layer called Beats that helps to collect OS level metrics as well as JVM and DB specific data.  There is an alerts module called X-Packs that we haven't yet started to look at yet.  Internally we were also building a common Elastic infrastructure so this path seemed to be reasonable for us to pursue for the moment.

But definitely agree that regardless of the tool, you need to have this type of correlating data visualization to understand the system performance.

[Vickey Palzor Lepcha]

@Alexey Matveev I'm having a tough time creating a datasource. Could you help me ?

I am doing everything on localhost.

I've selected Prometheus as a type. what should be my url ? It says default prometheus server url on 9090. But I have installed only plugin - not server.

[P_D_ Foerster]

Great article, thank you!

[Darrell Sutherland]

Fantastic Article! Cheers!

[Shivam Srivastava]

I agree it is a wonderful article.

Any expected time line when we can expect the feature of customizing the metrics as quoted in the article in Features section as-

"There are plans to let users define their own metircs, using the Groovy language. You will be able to collect your custom metrics not only from the Atlassian applications but also from other plugins. Also there will be metrics collected from Jira Service Desk."?

[amer]

Very nice, do you guys have any recommended Grafana dashboards asides from the stock ones at their site?

[Steffen Stundzig]

Hi @seniorandreymarkelov

thanks for this great plugins. I'll read the metrics with the existing telegraf plugin. But the metrics URL is secured and needs a user authentication in JIRA. In Confluence and Bitbucket, it's working great.

Do you know a way, to disable security for the metrics URL in JIRA?

Thanks in advance.

Regards

   Steffen...

[Richard Prigorodon]

Can we use this on data center?

[Alexey Matveev]

Yes, it can be used on Data Center, though there is not label for it.

[hmedia01]

No metrics for Jira Projects (Update Issues per Project)

Server edition (latest)

[Atif Baig]

Can someone setup this for us we are running jira on premises. 

[hmedia01]

Yes, i can. 

[Atif Baig]

Hi hmedia01 , please share your email address. I will touch base with you. 

[Brian Jones]

Trying to set this up and I am running into an issue @Alexey Matveev 

I Have the plugin installed in our Jira server, I used the docker install in the link above and i can get to Prometheus and Grafana. However when i add the line to the prometheus.yml file and restart Prometheus it seems like it never pulls in the Jira data. What am i doing wrong?

[Robert Banister]

Hope you don't mind me pasting my yml here but I am trying to configure for Jira Data center.  Prometheus and Grafana start up fine, however, when I go to the Targets page in Prometheus nothing is appearing.  I am hosting grafana and prometheus on node 1.  Where am I messing up?

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:

   - job_name: 'jira'
  scheme: http # change to http if don't you have https
  metrics_path: '/jira/plugins/servlet/prometheus/metrics'
    static_configs:
    - targets: ['node1_ip:8080', 'node2_ip:8080', 'node3_ip:8080'] # Jira host and port you serve
    labels:
        nodeId: node1
        ip: node1_ip
        nodeId: node2
        ip: node2_ip
        nodeId: node3
        ip: nod3_ip

   - job_name: 'grafana'
  scheme: http # change to http if don't you have https
  metrics_path: '/metrics'
    static_configs:
    - targets: ['node1_ip:3000'] # Jira host and port you serve

   - job_name: 'prometheus'
  scheme: http # change to http if don't you have https
  metrics_path: '/metrics'
    static_configs:
    - targets: ['node1_ip:9090'] # Jira host and port you serve

[Brian Jones]

Hey Robert,

I think you did the same thing as me. Under the Jira section here:

metrics_path: '/jira/plugins/servlet/

 Get rid of the /jira in that path. It's the context path for Jira. So if you don't use one then remove that.once I did that it started working.

[Robert Banister]

Hey Brian,

Thank you for the quick reply!

We do use the "/jira" context path for our instance (jira.site.com/jira is our base URL) so I do need to keep it, but I am still not seeing anything on the targets page (all/unhealthy) in Prometheus (http://myhost:9090/targets)

From the blog post: https://blog.servicerocket.com/adoption/how-to-monitor-atlassian-applications-performance-on-real-time-for-free this is what I should be seeing per my config, right?  Do I need to change the ownership of the top prometheus directory so everything is owned by root? 

[Robert Banister]

Hi Everyone,

I ended up opening a support case with cprime and they provided a corrected yml file.  Mine was overcomplicated to say the least.  Now I can see Targets in Prometheus and Grafana is reporting data as expected.  For context, node1 is where I have Prometheus and Grafana installed.  Hope this helps everyone:

# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).


# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
- job_name: 'jira'
scheme: http # change to http if don't you have https
metrics_path: '/jira/plugins/servlet/prometheus/metrics'
static_configs:
- targets: ['node1_ip:8081', 'node2_ip:8081', 'node3_ip:8081']

- job_name: 'grafana'
scheme: http # change to http if don't you have https
metrics_path: '/metrics'
static_configs:
- targets: ['node1_ip:3000']

- job_name: 'prometheus'
scheme: http # change to http if don't you have https
metrics_path: '/metrics'
static_configs:
- targets: ['node1_ip:9090']