OpsGenie Conditional alert silence

[Иван Поваляев]

Good day! 

We use Zabbix and Prometheus for monitoring. They are sending alerts to opsgenie and those alerts are routed to different Teams channels. So I was wondering can we silence alerts from different sources based on conditions (like name, tag etc.) in opsgenie.

E.g. Some server goes down - we receive Zabbix alert that it is indeed down and then Prometheus alerts that some services dependent on this server are failing. So the idea is to mute Prometheus alerts based on the condition that we already received alert from Zabbix. At this point it doesn't matter whether we receive them as alerts in ALERTS list, just not sending notification will do. Much like alert inhibition in Prometheus.

Thank you!

[John M · Accepted solution]

Hi @Иван Поваляев ,

You can silence alert notifications based on tags and other alert attributes using team-based notification policies.

The alerts must be assigned to the team where the policies are configured for notification policies to be applied. 

Keep in mind, this would only work if you could configure some way for only the alerts that you didn't want to notify to have those tags. There is no way to have conditions based on an existing alert.  

So, for example, you wouldn't be able to check your open alerts to see if an open Zabbix alert exists before deciding whether to silence a Prometheus alert whose service is dependent on that Zabbix server.

[Иван Поваляев]

Got it. Thank you very much.