Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Assistance creating Jira ticket as Prometheus alert fires

Hello 

I have a requirement where in a Jira ticket automatically needs to be created as Prometheus alert fires. As of now, we also have a setup where the alert gets received on Slack as well. 

I'm trying to use webhook url in the alertmanager configuration to use jira link. Below is how my alertmanager config looks like:

--------------------------- 

receivers:

- name: 'slack-general'
webhook_configs:
- url: "https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support"
slack_configs:
- api_url: "https://<>"
username: '{{ template "slack.default.username" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
title: '{{ template "slack.default.title" . }}'
title_link: '{{ template "slack.default.titlelink" . }}'
pretext: '{{ .CommonAnnotations.summary }}'
text: |-
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
• *Start Time:* {{ .StartsAt }}
{{ end }}
fallback: '{{ template "slack.default.fallback" . }}'
icon_emoji: '{{ template "slack.default.iconemoji" . }}'
icon_url: '{{ template "slack.default.iconurl" . }}'
#title: '{{ template "custom_title" . }}'
#text: '{{ template "custom_slack_message" . }}'
#text: '{{ template "email.instance.html" . }}'
- name: 'all-support'
email_configs:
- to: 'abc@abc.com'
html: '{{ template "email.instance.html" . }}'
headers:
subject: "ALERT!!: {{ .CommonLabels.instance}} | {{ .CommonLabels.job }} | {{ .CommonLabels.alertname }}"

templates:
- '/etc/prometheus/templates/instance.tpml'
- '/etc/alertmanager/template/*.tmpl'

 

But with this setup I'm gettting 405 error as seen below:

alertmanager[4870]: level=error ts=2019-06-26T05:59:02.698618182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 405 from https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support"

 

I have referred the document  https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/#getting-metadata-for-creating-issues-examples and from what I see it requires POST body with 'input' fields to be entered. Wondering how do i input these in the alertmanager config or anywhere else?. Can someone please guide me how to set this up.

 

Thanks

 

 

2 answers

1 accepted

0 votes
Answer accepted
Dario B Atlassian Team Jun 27, 2019

Hi @Ashma_Shriyan ,

I don't know how Prometheus alertmanager configuration works and what is the right syntax. However, if you are trying to have tickets  created in Jira once an alert is fired, then the easiest way to achieve this would be the following:

  1. Configure Prometheus to send an email to Jira when an alert fires
  2. In Jira create a user having as email address the one Prometheus uses to send emails (or configure a default reporter as explained in the documentation page in the next step)
  3. Create an email handler that will create tickets from these emails in the desired project in Jira as explained in the following documentation page: Creating issues and comments from email

 

In case this is not helping, or you are trying to do something different then please provide more details and/or an example.

 

Finally, I am tagging @Alexey Matveev in this thread since he is the one that wrote the below article and he may be able to help in here (also, review the below page, you may find it interesting):

 

I hope this helps.

 

Cheers,
Dario

Hi Dario,

Thanks for your response. 

I liked the solution that you proposed. I will go with it if I'm unable to achieve what  I want with my current setup.

I understand that you are not familiar with Prometheus setup but to define briefly, I have been following https://github.com/free/jiralert/blob/master/README.md to create automated ticket in Jira as Prometheus alert triggers. The current status is that, the test command mentioned below (as per the link) returns no error and a ticket gets created in Jira but when an alert is triggered in Prometheus there is no ticket created and an error "level=error ts=2019-06-28T06:04:44.252355609Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="context deadline exceeded" is logged, which is where I'm stuck.

 curl -H "Content-type: application/json" -X POST \
-d '{"receiver": "slack-general", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' \
http://localhost:9097/alert  

Thanks!

Dario B Atlassian Team Jun 28, 2019

Hi @Ashma_Shriyan ,

If I understand correctly:

  1. If you simulate that an alert was fired by calling the  below curl command everything works fine:
    curl -H "Content-type: application/json" -X POST -d '{"receiver": "slack-general", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' http://localhost:9097/alert  
  2. An error is returned when a real alert is fired in Prometheus.

 

If this is correct, then the issue seems to be that JiraAlert either did not receive a webhook from Prometheus at all, or it received a wrong one. 

 

Now, having a quick look at the below documentation page, it looks like you are setting the wrong url for the WebHook receiver:  

 

Indeed, when simulating the WebHook call by using curl you are sending the POST request to: 

  • http://localhost:9097/alert

While in your configuration file I can see  that the url for the webhook receiver is set to:  

 

 

To summarize: in Prometheus WebHook URL, you have to configure the URL to reach the machine on which JiraAlert is running (localhost is only fine if they are both running on the same machine) and not the one to reach Jira. 
Something like:

  • http://MACHINE_RUNNING_JIRALERT:9097/alert

Please make sure that the url is reachable from the machine where Prometheus is running.

 

Finally, please notice that neither jiralert nor Prometheus are Atlassian products and in case you need further help it might be a good idea to get in touch with Prometheus community as explained in: https://prometheus.io/community/

 

Have a nice weekend

 

Dario

Hey,

We're working with ZigiWave for some time. They're having a large number of integrations, so you can try reaching out to them for help. Their specialists and support team are extremely good. Their website is https://zigiwave.com/jira-software/ and might be able to help.

Regards,
Brad

Hey guys, apologies on the delay getting back to this. I was successfully able to configure jiralert and the ticket is also getting created as alert fires. 

Thanks all for your help and suggestion.

Like Dario B likes this
Dario B Atlassian Team Jul 12, 2019

@Ash  Just to know, was it the WebHooks URL in the configuration or anything else?  :) 

Have a nice weekend!

yes, it was just the incorrect webhook url :) 

Like Dario B likes this

@Ashma_Shriyan I am also trying to implement Automatic jira ticket creation for critical prometheus alert.
could you please help me in that what is the exact procedure to follow?

Like Dario B likes this

@Ashma_Shriyan I found some bugs in jiralert tool and I feel ,it is not secured one.
Do you have any other idea of creating jira ticket automatically other than jiralert and email handler method.

Like Richard Cross likes this

Hi,

When i run this:

curl -H "Content-type: application/json" -X POST -d '{"receiver": "jira-ab", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' http://localhost:9097/alert

I get an error about "unsupported value type" :

level=error ts=2020-01-09T01:34:49.698466622Z caller=main.go:143 msg="error handling request" statusCode=500 statusText="Internal Server Error" err="JIRA request https://myjira.mycompany.com/rest/api/2/search?jql=project%3D%22MYPROJECT%22+and+labels%3D%22ALERT%7Balertname%3D%5C%22TestAlert%5C%22%7D%22+order+by+resolutiondate+desc&maxResults=2&fields=summary,status,resolution,resolutiondate returned status 400 , body \"\"" receiver=jira-ab groupLabels="unsupported value type"

What am i doing wrong?

Thanks

--Andrew

Dario B Atlassian Team Jan 10, 2020

Hi @Andrew_L ,

Error 400 (bad request) means that something is wrong with the body of the request. 

Specifically, what is returning error 400 is the below REST API GET request to the /rest/api/2/search endpoint:

 

Looking at the above url, I strongly believe that what is causing the error is: ALERT{alertname=\"TestAlert\"} 

In order to whether confirm or deny, you can just copy paste the full URL (above) into your web browser (assuming that you can access myjira.mycompany.com from there) and see if the request is correctly executed or if it returns the same error, with and without the offending bit.

 

Cheers,
Dario

Thanks, Dario.

Yes... The same exact API url produces a HTTP 500 in my web browser.

If i run the query via the Jira UI (https://myjira.mycompany.com/issues/?jql=.....),
I don't get any errors (but the search yields no records)

Does this mean my version of Jira (i'm running v7.4.4) doesn't support that query (ALERT{alertname=\"TestAlert\"} ?

Then what Jira version is supported?

Thanks

--Andrew

Dario B Atlassian Team Jan 14, 2020

Hi @Andrew_L ,

Can you please clarify below points:

  1. Do you have some issues in your Jira having label: ALERT{alertname="TestAlert"} ?
  2. If my understanding is correct, the same search query works fine if called from the UI (https://myjira.mycompany.com/issues/?jql=.....) and fails returning error 500 if called using the REST API endpoint: (https://myjira.mycompany.com/rest/api/2/search?jql=....).
    This is odd since the JQL should work the same way, regardless the way you are calling is (UI or REST API). So, if my understanding is the correct, please paste the URL that works and the one that does not work in your reply. Provide more details otherwise.

 

Cheers,
Dario

@AndrewL / @Dario B 

Bear in mind that Jiralert has not seen a release in nearly a year.  Additionally, the documentation seems to imply that it's designed for Jira Cloud, rather than Jira Server/Data Center.

I'm not getting the exact same issue as Andrew, but Jiralert is failing to re-open existing closed tickets... even though it can correctly avoid duplicating tickets in Open and In Progress, as well as closed tickets with the "Won't Fix" resolution. 

To be honest, you're better off writing your own webhook for this.

Dario B Atlassian Team Jan 22, 2020

 

Looking at Jiralert GitHub issues I can see that the last thread, from November 2019, is proposing to move to a different project (prometheus-community):

Not sure if there is any progress since then. 

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Jira Software

How to create Jira issus from Excel file?

When to use CSV importer When managing your processes in Jira, there are many occasions where you need to create a lot of tasks. Creating them one by one will cost you a lot of time and effort and i...

4,430 views 22 32
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you