Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,467,272
Community Members
 
Community Events
177
Community Groups

How to debug the reason for an alert priority change

We do have Prometheus integration with OpsGenie, alerts are forwarded fine. The grouping includes the "severity" label which is in turn used (GroupLabels) to map to the OpsGenie priority. When I check the activity log I see 

```
Alert created via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[65e0c9dd-fdc9-45d7-b494-004be77fb454] with customSource[https://alertmanager-main-openshift-monitoring.apps.s01.cmpf.schwarz/#/alerts?receiver=opsgenie] with tiny id [3004] id [7b99cf08-c715-4b25-aa4b-e1cb5208b13d-1623653042318]
```
following by 

```
Alert priority has been updated to P3 via Prometheus[S01 STACKIT_-_OCP_Prometheus-Create Alert] with incomingDataId[d9942f8c-e722-4dab-885b-1723e6c88991] with customSource[https://alertmanager-main-openshift-monitoring.apps.s01.cmpf.schwarz/#/alerts?receiver=opsgenie] (Previous priority was P5). Will stop ongoing escalations and notification rule steps, and restart the notification flow as if the alert is newly created now.
```
both events reported in the same minute. I have no idea why prio is updated to P3 and would like to debug that. Any hints or ideas? Did I configure sth. wrong?

3 answers

1 vote
Nick H Atlassian Team Jun 16, 2021

Hi  @Andreas Furbach ,

What seems to be happening is the alert is being deduplicated. This occurs when an alert's alias matches the alias of an open existing Opsgenie alert: https://support.atlassian.com/opsgenie/docs/what-is-alert-de-duplication/

When an alert is deduplicated with a new alert of higher priority, the deduplicated alert’s priority is increased, and its notification flow is restarted. This seems to be what's happening with your case: https://support.atlassian.com/opsgenie/docs/update-alert-priority-level/

You can review what data/fields are being parsed in the alias field under the integration's Advance tab:

promcomm1.jpg

 

By default it's the {{alias}} field - which will parse whatever data/fields determined by the configuration in Alert Manager - along with "grouping includes the 'severity' label which is in turn used (GroupLabels) to map to the OpsGenie priority."

Hope this helps! Let us know if you have any other questions, issues, etc.

1 vote

Hi Andreas,

If you search the incomingDataId in quotes in the logs (Settings > Logs), you can see the payload Opsgenie received from Prometheus for the priority change.

Thanks a lot for you answers, indeed it was deduplication because of a configuration mistake of mine. I overlooked a dash "- " at a wrong place in my alertmanager config so I had two channels opened to opsgenie. And the last one was then missing all my sophisticed "severity" to "priority" mapping. 

Thanks for your answers, very much appreciated. I'll dig into the "logs" bit in the setting, I cannot access those logs but I asked our admin to check. For future use ... :)

@Andreas Furbach Hi guys, What do you change into alertmanager config ? i afraid i have the same problem only P3 alerts devilered on OpsGenie

I had a problem with my config (alertmanager), here the wrong config, see the dash before responders. This introduced a second channel to the same opsgenie target without proper mapping of severities, thus ending up in P3. Deduplication did the rest. 

 

receivers:
- name: opsgenie
opsgenie_configs:
- send_resolved: true
api_url: 'https://api.eu.opsgenie.com/'
http_config:
proxy_url: http://http-proxy.schwarz:8080
tags: '{{ range .Alerts }}{{ .Labels.Values | join "," }}{{end}}'
details:
cluster: '{{ ocp_name }}'
priority: '{{ if eq .GroupLabels.severity "fatal" }}P1{{ else }}{{ if eq .GroupLabels.severity "critical" }}P2{{ else }}{{ if eq .GroupLabels.severity "warning" }}P3{{ else }}{{ if eq .GroupLabels.severity "low" }}P4{{ else }}{{ if eq .GroupLabels.severity "info" }}P5{{ else }}P5{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}'
- responders:
- name: OCP-TEAM
type: team

Thanks i'm fix my problem too by switching "- priority" to "priority" section into yaml config

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events