Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to debug the reason for an alert priority change

We do have Prometheus integration with OpsGenie, alerts are forwarded fine. The grouping includes the "severity" label which is in turn used (GroupLabels) to map to the OpsGenie priority. When I check the activity log I see 

```
Alert created via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[65e0c9dd-fdc9-45d7-b494-004be77fb454] with customSource[https://alertmanager-main-openshift-monitoring.apps.s01.cmpf.schwarz/#/alerts?receiver=opsgenie] with tiny id [3004] id [7b99cf08-c715-4b25-aa4b-e1cb5208b13d-1623653042318]
```
following by 

```
Alert priority has been updated to P3 via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[d9942f8c-e722-4dab-885b-1723e6c88991] with customSource[https://alertmanager-main-openshift-monitoring.apps.s01.cmpf.schwarz/#/alerts?receiver=opsgenie] (Previous priority was P5). Will stop ongoing escalations and notification rule steps, and restart the notification flow as if the alert is newly created now.
```
both events reported in the same minute. I have no idea why prio is updated to P3 and would like to debug that. Any hints or ideas? Did I configure sth. wrong?

3 answers

1 vote

Hi Andreas,

If you search the incomingDataId in quotes in the logs (Settings > Logs), you can see the payload Opsgenie received from Prometheus for the priority change.

1 vote
Nick H Atlassian Team Jun 16, 2021

Hi  @Andreas Furbach ,

What seems to be happening is the alert is being deduplicated. This occurs when an alert's alias matches the alias of an open existing Opsgenie alert: https://support.atlassian.com/opsgenie/docs/what-is-alert-de-duplication/

When an alert is deduplicated with a new alert of higher priority, the deduplicated alert’s priority is increased, and its notification flow is restarted. This seems to be what's happening with your case: https://support.atlassian.com/opsgenie/docs/update-alert-priority-level/

You can review what data/fields are being parsed in the alias field under the integration's Advance tab:

promcomm1.jpg

 

By default it's the {{alias}} field - which will parse whatever data/fields determined by the configuration in Alert Manager - along with "grouping includes the 'severity' label which is in turn used (GroupLabels) to map to the OpsGenie priority."

Hope this helps! Let us know if you have any other questions, issues, etc.

0 votes

Thanks a lot for you answers, indeed it was deduplication because of a configuration mistake of mine. I overlooked a dash "- " at a wrong place in my alertmanager config so I had two channels opened to opsgenie. And the last one was then missing all my sophisticed "severity" to "priority" mapping. 

Thanks for your answers, very much appreciated. I'll dig into the "logs" bit in the setting, I cannot access those logs but I asked our admin to check. For future use ... :)

@Andreas Furbach Hi guys, What do you change into alertmanager config ? i afraid i have the same problem only P3 alerts devilered on OpsGenie

I had a problem with my config (alertmanager), here the wrong config, see the dash before responders. This introduced a second channel to the same opsgenie target without proper mapping of severities, thus ending up in P3. Deduplication did the rest. 

 

receivers:
- name: opsgenie
opsgenie_configs:
- send_resolved: true
api_url: 'https://api.eu.opsgenie.com/'
http_config:
proxy_url: http://http-proxy.schwarz:8080
tags: '{{ range .Alerts }}{{ .Labels.Values | join "," }}{{end}}'
details:
cluster: '{{ ocp_name }}'
priority: '{{ if eq .GroupLabels.severity "fatal" }}P1{{ else }}{{ if eq .GroupLabels.severity "critical" }}P2{{ else }}{{ if eq .GroupLabels.severity "warning" }}P3{{ else }}{{ if eq .GroupLabels.severity "low" }}P4{{ else }}{{ if eq .GroupLabels.severity "info" }}P5{{ else }}P5{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}'
- responders:
- name: OCP-TEAM
type: team

Thanks i'm fix my problem too by switching "- priority" to "priority" section into yaml config

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Opsgenie

Leveraging Atlassian’s Opsgenie and AWS Cloudformation Registry to stay ahead of incidents

In the past, managing IT infrastructure was a hard job. It required a lot of manual effort and it was hard to keep track of all the necessary information (monitoring, scalability etc). Thankfully, as...

973 views 0 9
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you