How to debug the reason for an alert priority change

[Andreas Furbach]

We do have Prometheus integration with OpsGenie, alerts are forwarded fine. The grouping includes the "severity" label which is in turn used (GroupLabels) to map to the OpsGenie priority. When I check the activity log I see 

```
Alert created via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[65e0c9dd-fdc9-45d7-b494-004be77fb454] with customSource[https://alertmanager-main-openshift-monitoring.apps.s01.cmpf.schwarz/#/alerts?receiver=opsgenie] with tiny id [3004] id [7b99cf08-c715-4b25-aa4b-e1cb5208b13d-1623653042318]
```
following by 

```
Alert priority has been updated to P3 via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[d9942f8c-e722-4dab-885b-1723e6c88991] with customSource[https://alertmanager-main-openshift-monitoring.apps.s01.cmpf.schwarz/#/alerts?receiver=opsgenie] (Previous priority was P5). Will stop ongoing escalations and notification rule steps, and restart the notification flow as if the alert is newly created now.
```
both events reported in the same minute. I have no idea why prio is updated to P3 and would like to debug that. Any hints or ideas? Did I configure sth. wrong?

[Nick H]

Hi  @Andreas Furbach ,

What seems to be happening is the alert is being deduplicated. This occurs when an alert's alias matches the alias of an open existing Opsgenie alert: https://support.atlassian.com/opsgenie/docs/what-is-alert-de-duplication/

When an alert is deduplicated with a new alert of higher priority, the deduplicated alert’s priority is increased, and its notification flow is restarted. This seems to be what's happening with your case: https://support.atlassian.com/opsgenie/docs/update-alert-priority-level/

You can review what data/fields are being parsed in the alias field under the integration's Advance tab:

By default it's the {{alias}} field - which will parse whatever data/fields determined by the configuration in Alert Manager - along with "grouping includes the 'severity' label which is in turn used (GroupLabels) to map to the OpsGenie priority."

Hope this helps! Let us know if you have any other questions, issues, etc.

[Brennan Kiely]

Hi Andreas,

If you search the incomingDataId in quotes in the logs (Settings > Logs), you can see the payload Opsgenie received from Prometheus for the priority change.

[Andreas Furbach]

Thanks a lot for you answers, indeed it was deduplication because of a configuration mistake of mine. I overlooked a dash "- " at a wrong place in my alertmanager config so I had two channels opened to opsgenie. And the last one was then missing all my sophisticed "severity" to "priority" mapping. 

Thanks for your answers, very much appreciated. I'll dig into the "logs" bit in the setting, I cannot access those logs but I asked our admin to check. For future use ... :)

[vitalii_melnyk]

@Andreas Furbach Hi guys, What do you change into alertmanager config ? i afraid i have the same problem only P3 alerts devilered on OpsGenie

[Andreas Furbach]

I had a problem with my config (alertmanager), here the wrong config, see the dash before responders. This introduced a second channel to the same opsgenie target without proper mapping of severities, thus ending up in P3. Deduplication did the rest. 

receivers:
  - name: opsgenie
    opsgenie_configs:
    - send_resolved: true
    api_url: 'https://api.eu.opsgenie.com/'
    http_config:
    proxy_url: http://http-proxy.schwarz:8080 
    tags: '{{ range .Alerts }}{{ .Labels.Values | join "," }}{{end}}'
    details:
      cluster: '{{ ocp_name }}' 
    priority: '{{ if eq .GroupLabels.severity "fatal" }}P1{{ else }}{{ if eq .GroupLabels.severity "critical" }}P2{{ else }}{{ if eq .GroupLabels.severity "warning" }}P3{{ else }}{{ if eq .GroupLabels.severity "low" }}P4{{ else }}{{ if eq .GroupLabels.severity "info" }}P5{{ else }}P5{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}'
    - responders:
      - name: OCP-TEAM
        type: team

[vitalii_melnyk]

Thanks i'm fix my problem too by switching "- priority" to "priority" section into yaml config