Notification Policy - Deduplication Count or Time Delay

Jake Houser

Hello,

I have a bit of a complex question and was curious if anyone here might have some better insight on how to achieve this:

My goal is that if am alert comes in at night, a user should not receive a notification until the next morning, unless an alert with that same alias comes in multiple times. At that point, I'd like to push the notification through or increase its priority to trigger a different notification flow. However, from what I can see in the documentation only one notification policy will be applied so I can either suppress the notification until a specific time or until a deduplication count but not both.

 

I'm open to other methods of solving this problem, which basically boils down to: if one server in a group triggers an alert, it's not necessarily critical, but if multiple servers are all having the same problem, I'd like to aggregate those alerts into a higher priority one. In this example, all of the alerts are triggered individually (from Kapacitor) so they don't know about each other. Is there a way I could manage this other than trying to rely on notification policies? Should I just trigger an auto-restart with a long enough time delay to fire the alert again and hope the time-based notification policy no longer applies?

 

Any help here would be appreciated!

Samir

Hi @Jake Houser !

 

You bring up a good point, it's not currently possible to be able to delay notifications until next morning unless the deduplication count reaches a certain number. You're right that it's either, delay notifications until next morning OR delay notifications until deduplication count reaches a certain number. There isn't a way to do both.

 

We do have a feature request to be able to adjust the priority of the alert based on the deduplication count using an Alert policy. So if that was an option, you could just use a notification policy to delay all alerts until next morning if priority is less than P1 for example. And then setup an alert policy to increase the priority of the alert if the deduplication count reaches a certain number. So in that case, notifications would be delayed till next morning unless the deduplication count reaches a certain number, in which case the priority of the alert would increase and the notifications would trigger right away.

 

I will add this use-case to that feature request. Thanks for the input!