nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Controlling (the Flood) of Notifications in OpsGenie

colin.schofield

We have recently replaced Splunk On-Call (formerly known as VictorOps) with a new setup of OpsGenie.

I am a user/admin of OpsGenie and we have a really annoying problem to do with 'Notification fatigue'.

The scenario is as follows: We receive a repeated alert to do with a problem in the system and this message floods our slack channel and the person on-call is bombarded with the notifications.

We would like to have the following happen:

1. On-Call receive the alert (and a message is sent to the slack channel)
2. On-Call Acknowledges the alert
3. Unless they close the alert, they will not receive any more of the same notification again.

I have taken a look at the Current Tier->Policies->Notification Policies, but I cannot see this feature.

Also, this is exactly how Splunk On-Call worked..

Does anyone have any ideas how to fix it or perhaps work around?

(btw, I have looked at the video/document called 'Three Ways to Reduce Alert Noise and Fatigue with Opsgenie')

Accepted answers

Tom Russell

The alias alert field is your best friend here. Make sure to set that to a value that uniquely identifies that series of alerts, or that problem. You'll get a notification for the first alert, but then each subsequent alert will just increase the counter on the first alert, without notifying. If you close the alert in Opsgenie and the problem is still happening, it will open a new alert, starting the counter again.

All comments

Tom Russell

colin.schofield

@Tom Russell This sounds like exactly what I need!!

Are there any instructions, documentation, blog, tutorial or youtube that covers this feature in-depth?

Thanks!

colin.schofield

@Tom Russell

As a follow-up, this did exactly what I needed -- thank you very much for the advice!

I found that the Alias was coming in as a UID which was always different, so what I did was I created a settings->Global Policies that set the Alias to equal the message..

It works now 👍

Here are a few images that others may find useful.

Tom Russell

@colin.schofield, one thing to be careful of: we usually imbed the date somewhere in the alias. That way, if the alert gets acknowledged, then forgotten about, it will open a new one the next day. Otherwise, your problem could go for weeks without opening a new alert.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

opsgenie

alert