Deduping Notification Policy

[Craig Monroe]

Hi, 

In our notification policy workflow, I added policy that would dedupe for the same alias. 

For testing purposes, I set: 

- Match All Alerts

Then

- Delay/Suppress

-- Delay notifications unless the de-duplication count is equal to "5"

I then sent test alerts in via datadog

I see the count for the alert going up, but expected the after 5, that another alert notification would show in opsgenie. However, the count just continues to climb. 

Wondering what I am doing wrong. 

-----------------------------------------------------------------------------------------------

Update - I see in activity log that with both "Delay notifications unless de-duplication count is equal to"  

OR

"Delay notifications unless the alert is occurred 2 times in a time interval..."

that it is delaying the first notification.

Now that I read the rule again, that seems to do what it's saying.

But how do we receive a notification for the first alert, then de-dupe subsequent alerts"

The goal is for the team members to receive the initial alert, then suppress subsequent so they aren't overwhelmed for the same alert unless a threshold is breached. 

Thanks for any help,

Craig

[Andrew Laden · Accepted solution]

As long as you are de-duplicating alerts, the a user will never process a notification request twice for the same alert. (with one exception - See below)

OpsGenie is all about expecting someone to respond to an alert when it happens. Once a person is notified, If that person doesn't respond, the OG follows the escalation rules.

1. Opsgenie receives a message. It runs through the integration rules to decide if it is a new alert, a duplicate, or if it can be ignored.
2. If it is a new alert, it creates the alert, then applies policies to the alert that could modify it, or affect notification. (such as a delay until x repeats policy)
3. If it determines that a notification should be sent. it passes to the team. The team configuration will process the routing rules to determine what escalation policy to use.
4. The escalation policy will determine who to notify and when.
5. When the escalation policy says to notify, it goes to the user object and tells it to notify
   1. If the user has already been notified in a previous step, they are not notified again. If they didn't respond the first time, why would they the 2nd time.
6. The user object then processes its notification policies to determine HOW to notify the user, an the notifies the user following the notification rules  in that policie. 
   1. At this stage, a user can set up multiple levels of notification to themselves.
   2. Notification rules and be set to repeat and renotify, but that is a per user setting. (unless you are using enterprise notification policies.

So what this means:

1: OG will never send a 2nd "request to notify" to a user for the same alert as long as the alert is de-duplicated. 1 alert = 1 notification

2: A user will can not be added to multiple levels of an escalation to get re-notified. one the are notified thats it.

The design is that it expects people to respond to notifications. If they don't it moves on.

So there isnt a way to say "Notify me initially, and then notify me again if the de-duplication count reaches x" Its all 1 alert. Either notify me when it first happens, or notify me when the duplication count reaches x. The expectation is that until the notification is sent out, no response is needed. But when it is sent out, someone needs to respond. there is no "warn me about this, but dont care if i dont respond unless something else happens"

The exception. an escalation can be configured such that if it reaches the end of the escalation policy, and you have it set to repeat the escalation and reset the seen and acknowledged states, then it will clear everything and re-notify. But that has nothing to do with duplication counts. Its just time based.

[Craig Monroe]

Thanks Andrew. I appreciate the clarification.