How to set Opsgenie alert priority from Prometheus alert priority

Ashma Shriyan October 24, 2019

Hello

I have the priority level in Prometheus set as 'severity' with values low,medium and critical. I tried using Opsgenie's 'Custom Value to Opsgenie priority' option with value {{severity.substring(0,2)}} but that does not seem to change the priority level of the Opsgenie alert. What am I missing here? Can someone please help me.

Thanks!

4 answers

Comments for this post are closed

Community moderators have prevented the ability to post new answers.

Post a new question

1 vote
Shiv Pathak August 13, 2020

Hi Folks,

 

Posting my configuration which works well with severity mapping to priority in opsgenie as follows:

 

critical => P1

warning => P2

info => P3

none => P4

 

Configuration for Alertmanager I used as follows:

global:
resolve_timeout: 1m
opsgenie_api_url: https://api.eu.opsgenie.com/
opsgenie_api_key: <my-top-secret-api-key>
receivers:
- opsgenie_configs:
- teams: <my-team-name>
priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else if eq .Labels.severity "warning"}}P2{{else if eq .Labels.severity "info"}}P3{{else}}P4{{end}}{{end}}'
name: opsgenie
route:
group_by: ['...']
receiver: opsgenie
repeat_interval: 5m

I tested this with Opsgenie free tier with Prometheus Alertmanager.

 

Please mind indentations while configuring.

 

Final Note: please input api-key, team name accordingly, also group_by etc fields are not optimised, I tested this with bare 4 alerts having specific severity what I mentioned above.

Prabhu Thangaraj May 24, 2021
global:
resolve_timeout: 1m
opsgenie_api_url: https://api.eu.opsgenie.com/
opsgenie_api_key: "XXXXXXXXXX"
receivers:
- opsgenie_configs:
- responders:
- type: team
name: "<team name>"
priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else if eq .Labels.severity "warning"}}P2{{else if eq .Labels.severity "info"}}P3{{else}}P4{{end}}{{end}}'
name: opsgenie
route:
group_by: ['alertname','service']
receiver: opsgenie
repeat_interval: 5m
Prabhu Thangaraj May 24, 2021

The above configuration worked for me i was working with a Atlassian Support, and we tried the above config it worked for us

Jason Walton March 9, 2022

The above will work, but only if if you only have one alert!  The problem is this line:

priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else if eq .Labels.severity "warning"}}P2{{else if eq .Labels.severity "info"}}P3{{else}}P4{{end}}{{end}}'


The "range" here will iterate over all alerts, so if you have more than one alert this will generate a priority like "P2P2", which opsgenie will reject:

level=error ts=2022-03-09T14:59:23.350Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="opsgenie/opsgenie[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 422: {\"message\":\"Request body is not processable. Please check the errors.\",\"errors\":{\"priority\":\"should be one of [ P1, P2, P3, P4, P5 ]\"},\"took\":0.001,\"requestId\":\"c4390ce3-f99c-4149-820d-1886421ef213\"}"


I'm using this configuration to group together alerts of the same priority and send them to opsgenie as a batch:

receivers:
- name: "null"
- name: "opsgenie - critical"
opsgenie_configs:
- priority: P1
- name: "opsgenie - warning"
opsgenie_configs:
- priority: P2
- name: "opsgenie"
opsgenie_configs:
- priority: P3
route:
receiver: "null"
group_by:
- job
routes:
# Alerts in this receiver are ignored.
- receiver: "null"
match:
alertname: Watchdog
- receiver: "opsgenie - critical"
match:
severity: "critical"
- receiver: "opsgenie - warning"
match:
severity: "warning"
- receiver: "opsgenie"
Like # people like this
1 vote
Daniel_Leite January 17, 2020

My prometheus+alertmanager work fine with priorities!

Example:

in prometheus, i have this alert:


- alert: Instance_Free_Inodes
expr: (disk_inodes_used * 100 / disk_inodes_total) > 75
for: 5m
labels:
severity: page
alert_target: '{{ $labels.host }}'
annotations:
description: '{{ $labels.alias }} of job {{ $labels.job }} ({{ $labels.path }}) has ({{ $value | printf "%.2f" }})% used inodes for more than 5 minutes.'
summary: {{ $labels.alias }} using more than 75% inodes on {{ $labels.path }}
impact: Services may fail
action: Check the server disks and remove some files, if possible
dashboard: https://grafana.example.com/d/pjhLJOzmk/infrastructure-hosts-stats-3hours
priority: P3

Then in alertmanager:


- name: opsgenie_techops_team
  opsgenie_configs:
    - send_resolved: true
      http_config: {}
      api_key: <secret>>
      api_url: https://api.opsgenie.com/
      message: '{{ template "opsgenie.default.message" . }}'
      description: '{{ template "opsgenie.default.description" . }}'
      source: '{{ template "opsgenie.default.source" . }}'
      responders:
      - name: techops_team
         type: team
      priority: '{{ if .CommonAnnotations.priority }}{{ .CommonAnnotations.priority
}}{{ else }}P3{{ end }}'

 

 

This way i can setup the priority in the prometheus alert and it is pushed to opsgenie correctly

0 votes
Nick H
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 12, 2019

With Prometheus' Alert Manager config file you should be able to map the severity field with something customer like this:

priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else}}P3{{end}}{{end}}'

You can always change "P1" "P3" to their preference.

image.png

Ashma Shriyan November 12, 2019

@Nick H Thanks! I tried that out and it maps  priority to P1 for critical alerts but I still wonder why does the one on Opsgenie portal did not work out. ie,  Setting 'Custom Value to Opsgenie priority' option with value {{severity.substring(0,2)}} 

Ashma Shriyan November 17, 2019

Any update on this yet?

Nick H
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 18, 2019

I'm realizing Prometheus unfortunately is not an integration that can have a custom value to Opsgenie property configured such as a value {{severity.substring(0,2)}}:

priority image.jpg

 

So the best option is mapping this on the Alert Manager side like shown above. We also see customers using Alert Policies to parsed for the severity field, and modify/map the alert's priority this way, but these policies are only included in our Standard and Enterprise plans: https://docs.opsgenie.com/docs/global-policies#section-alert-policy

Like # people like this
Ashma Shriyan November 18, 2019

apologies, but i'm not satisfied with your answer. If you say Prometheus is not an integration that can have a custom value to Opsgenie property, why is the option 'Custom Value to Opsgenie priority' even included in Prometheus integration page? If that cannot have values like '{{severity.substring(0,2)}}' what other values can it have? Can you please give me an example so that I understand the feature better?

Nick H
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 18, 2019

It's not an option (unless you are seeing something else with your integration):

priorty 2.jpg

Prometheus is using an API key to post to Opsgenie and because of this, looking for the values that are priority is (P1-5).

Ashma Shriyan November 18, 2019

 

This is what I see and I would like to know  how to use it and what values does it take.Opsgenie.png

Nick H
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 19, 2019

You can use this option (Custom Value to Opsgenie Priority) with other webhook-based integrations. Unfortunately it does not work with our API integrations such as Prometheus.

Prometheus is an integration that sends notifications through our API (https://prometheus.io/docs/alerting/configuration/#opsgenie_config) and due to this - Opsgenie will be looking for exactly this value being sent from the alert API.

So for your use-case, you'd need to map this in Alert Manager (since you are on a Free plan with limited functionality to other ways of mapping this in Opsgenie).

0 votes
Nick H
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 25, 2019

Hi Ashman,

I think you are on the right track, but missing one character. Can you try this string instead:

P{{severity.substring(0,2)}}

The priority is touchy in this sense that it needs a hardcoded 'P' to map to our priorities.

Let us know if this does the trick! If not, I may need to take a look at what the logs are parsing or what the severity looks like.

Ashma Shriyan October 26, 2019

@Nick H Thanks for the response. I tried what you suggested but that did not seem to work. Please take a look and let me know about the workaround. Thanks!

Comments for this post are closed

Community moderators have prevented the ability to post new answers.

Post a new question

TAGS
AUG Leaders

Atlassian Community Events