Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
Community Members
Community Events
Community Groups

How to debug the reason for an alert priority change

We do have Prometheus integration with OpsGenie, alerts are forwarded fine. The grouping includes the "severity" label which is in turn used (GroupLabels) to map to the OpsGenie priority. When I check the activity log I see 

Alert created via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[65e0c9dd-fdc9-45d7-b494-004be77fb454] with customSource[] with tiny id [3004] id [7b99cf08-c715-4b25-aa4b-e1cb5208b13d-1623653042318]
following by 

Alert priority has been updated to P3 via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[d9942f8c-e722-4dab-885b-1723e6c88991] with customSource[] (Previous priority was P5). Will stop ongoing escalations and notification rule steps, and restart the notification flow as if the alert is newly created now.
both events reported in the same minute. I have no idea why prio is updated to P3 and would like to debug that. Any hints or ideas? Did I configure sth. wrong?

3 answers

1 vote
Nick H Atlassian Team Jun 16, 2021

Hi  @Andreas Furbach ,

What seems to be happening is the alert is being deduplicated. This occurs when an alert's alias matches the alias of an open existing Opsgenie alert:

When an alert is deduplicated with a new alert of higher priority, the deduplicated alert’s priority is increased, and its notification flow is restarted. This seems to be what's happening with your case:

You can review what data/fields are being parsed in the alias field under the integration's Advance tab:



By default it's the {{alias}} field - which will parse whatever data/fields determined by the configuration in Alert Manager - along with "grouping includes the 'severity' label which is in turn used (GroupLabels) to map to the OpsGenie priority."

Hope this helps! Let us know if you have any other questions, issues, etc.

1 vote

Hi Andreas,

If you search the incomingDataId in quotes in the logs (Settings > Logs), you can see the payload Opsgenie received from Prometheus for the priority change.

Thanks a lot for you answers, indeed it was deduplication because of a configuration mistake of mine. I overlooked a dash "- " at a wrong place in my alertmanager config so I had two channels opened to opsgenie. And the last one was then missing all my sophisticed "severity" to "priority" mapping. 

Thanks for your answers, very much appreciated. I'll dig into the "logs" bit in the setting, I cannot access those logs but I asked our admin to check. For future use ... :)

@Andreas Furbach Hi guys, What do you change into alertmanager config ? i afraid i have the same problem only P3 alerts devilered on OpsGenie

I had a problem with my config (alertmanager), here the wrong config, see the dash before responders. This introduced a second channel to the same opsgenie target without proper mapping of severities, thus ending up in P3. Deduplication did the rest. 


- name: opsgenie
- send_resolved: true
api_url: ''
tags: '{{ range .Alerts }}{{ .Labels.Values | join "," }}{{end}}'
cluster: '{{ ocp_name }}'
priority: '{{ if eq .GroupLabels.severity "fatal" }}P1{{ else }}{{ if eq .GroupLabels.severity "critical" }}P2{{ else }}{{ if eq .GroupLabels.severity "warning" }}P3{{ else }}{{ if eq .GroupLabels.severity "low" }}P4{{ else }}{{ if eq .GroupLabels.severity "info" }}P5{{ else }}P5{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}'
- responders:
- name: OCP-TEAM
type: team

Thanks i'm fix my problem too by switching "- priority" to "priority" section into yaml config

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted in Jira Service Management

Jira Service Management Documentation Opportunities

Hello everyone, Hope everyone is safe! A few months ago we posted an article sharing all the new articles and documentation that we, the AMER Jira Service Management team created. As mentioned ...

330 views 0 10
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you