nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Assistance creating Jira ticket as Prometheus alert fires

Ashma Shriyan

Hello

I have a requirement where in a Jira ticket automatically needs to be created as Prometheus alert fires. As of now, we also have a setup where the alert gets received on Slack as well.

I'm trying to use webhook url in the alertmanager configuration to use jira link. Below is how my alertmanager config looks like:

---------------------------

receivers:

- name: 'slack-general'
webhook_configs:
- url: "https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support"
slack_configs:
- api_url: "https://<>"
username: '{{ template "slack.default.username" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
title: '{{ template "slack.default.title" . }}'
title_link: '{{ template "slack.default.titlelink" . }}'
pretext: '{{ .CommonAnnotations.summary }}'
text: |-
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
• *Start Time:* {{ .StartsAt }}
{{ end }}
fallback: '{{ template "slack.default.fallback" . }}'
icon_emoji: '{{ template "slack.default.iconemoji" . }}'
icon_url: '{{ template "slack.default.iconurl" . }}'
#title: '{{ template "custom_title" . }}'
#text: '{{ template "custom_slack_message" . }}'
#text: '{{ template "email.instance.html" . }}'
- name: 'all-support'
email_configs:
- to: 'abc@abc.com'
html: '{{ template "email.instance.html" . }}'
headers:
subject: "ALERT!!: {{ .CommonLabels.instance}} | {{ .CommonLabels.job }} | {{ .CommonLabels.alertname }}"

templates:
- '/etc/prometheus/templates/instance.tpml'
- '/etc/alertmanager/template/*.tmpl'

But with this setup I'm gettting 405 error as seen below:

alertmanager[4870]: level=error ts=2019-06-26T05:59:02.698618182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 405 from https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support"

I have referred the document https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/#getting-metadata-for-creating-issues-examples and from what I see it requires POST body with 'input' fields to be entered. Wondering how do i input these in the alertmanager config or anywhere else?. Can someone please guide me how to set this up.

Thanks

Accepted answers

Dario B

Hi @Ashma Shriyan ,

I don't know how Prometheus alertmanager configuration works and what is the right syntax. However, if you are trying to have tickets created in Jira once an alert is fired, then the easiest way to achieve this would be the following:

Configure Prometheus to send an email to Jira when an alert fires
In Jira create a user having as email address the one Prometheus uses to send emails (or configure a default reporter as explained in the documentation page in the next step)
Create an email handler that will create tickets from these emails in the desired project in Jira as explained in the following documentation page: Creating issues and comments from email

In case this is not helping, or you are trying to do something different then please provide more details and/or an example.

Finally, I am tagging @Alexey Matveev in this thread since he is the one that wrote the below article and he may be able to help in here (also, review the below page, you may find it interesting):

Monitoring of Atlassian applications with Prometheus

I hope this helps.

Cheers,
Dario

All comments

Dario B

Hi @Ashma Shriyan ,

Configure Prometheus to send an email to Jira when an alert fires
In Jira create a user having as email address the one Prometheus uses to send emails (or configure a default reporter as explained in the documentation page in the next step)
Create an email handler that will create tickets from these emails in the desired project in Jira as explained in the following documentation page: Creating issues and comments from email

In case this is not helping, or you are trying to do something different then please provide more details and/or an example.

Finally, I am tagging @Alexey Matveev in this thread since he is the one that wrote the below article and he may be able to help in here (also, review the below page, you may find it interesting):

Monitoring of Atlassian applications with Prometheus

I hope this helps.

Cheers,
Dario

Ashma Shriyan

Hi Dario,

Thanks for your response.

I liked the solution that you proposed. I will go with it if I'm unable to achieve what I want with my current setup.

I understand that you are not familiar with Prometheus setup but to define briefly, I have been following https://github.com/free/jiralert/blob/master/README.md to create automated ticket in Jira as Prometheus alert triggers. The current status is that, the test command mentioned below (as per the link) returns no error and a ticket gets created in Jira but when an alert is triggered in Prometheus there is no ticket created and an error "level=error ts=2019-06-28T06:04:44.252355609Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="context deadline exceeded" is logged, which is where I'm stuck.

curl -H "Content-type: application/json" -X POST \
-d '{"receiver": "slack-general", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' \
http://localhost:9097/alert

Thanks!

Dario B

Hi @Ashma Shriyan ,

If I understand correctly:

If you simulate that an alert was fired by calling the below curl command everything works fine:

curl -H "Content-type: application/json" -X POST -d '{"receiver": "slack-general", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' http://localhost:9097/alert

An error is returned when a real alert is fired in Prometheus.

If this is correct, then the issue seems to be that JiraAlert either did not receive a webhook from Prometheus at all, or it received a wrong one.

Now, having a quick look at the below documentation page, it looks like you are setting the wrong url for the WebHook receiver:

https://prometheus.io/docs/alerting/configuration/#webhook_config

Indeed, when simulating the WebHook call by using curl you are sending the POST request to:

http://localhost:9097/alert

While in your configuration file I can see that the url for the webhook receiver is set to:

https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support (this looks like an example url, I assume in the config you were having your real Jira url).

To summarize: in Prometheus WebHook URL, you have to configure the URL to reach the machine on which JiraAlert is running (localhost is only fine if they are both running on the same machine) and not the one to reach Jira.
Something like:

http://MACHINE_RUNNING_JIRALERT:9097/alert

Please make sure that the url is reachable from the machine where Prometheus is running.

Finally, please notice that neither jiralert nor Prometheus are Atlassian products and in case you need further help it might be a good idea to get in touch with Prometheus community as explained in: https://prometheus.io/community/

Have a nice weekend

Dario

Brad Collins_zb381e

Hey,

We're working with ZigiWave for some time. They're having a large number of integrations, so you can try reaching out to them for help. Their specialists and support team are extremely good. Their website is https://zigiwave.com/jira-software/ and might be able to help.

Regards,
Brad

evasheeva

Hey guys, apologies on the delay getting back to this. I was successfully able to configure jiralert and the ticket is also getting created as alert fires.

Thanks all for your help and suggestion.

Dario B

@evasheeva Just to know, was it the WebHooks URL in the configuration or anything else?

Have a nice weekend!

evasheeva

yes, it was just the incorrect webhook url

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

jira

jira-software

integration