Can one alert be used to close other alerts ?

Question

I am using Kapacitor to generate alerts based on thresholds.  Usually Kapacitor will call the API to close an alert when the threshold is no longer breached, alerts are closed by the {{alias}} field.

There are occasions when Kapacitor will not close the alert, namely when it stops receiving data.  In these occasions I would like to generate a new alert that will close any open alerts that meet a criteria, say matching tags.

Is this possible ?

Thanks in advance.

Neil Benton · Answer

I have solved this using the following method.  In my monitoring script in Kapacitor I generate two alerts based on different events, but each alert is given the same identifying key, this means that in opsgenie they get the same alias so alert deduplication works.

The first alert is generated if my threshold is breached.  so if cpu>80.

the second alert is generated when we are receiving data so we get a warning the system is working alert.  This seems counter intuitive, but I set a tag on the alert IGNORE=true.  I have an ignore filter in my opsgenie integration that looks at this tag and it isn't processed.

When we stop receiving data the second alert flips to the OK state and a alert close event is generated, this does not have the IGNORE tag set and so opsgenie closes all alerts with the same alias, this includes the generated thresholds.

It is used as we are monitoring servers in a scale set in the cloud, so these servers can and do disappear at random.

Neil Benton · Answer

Thanks for the suggestions

connor mcguane · Answer

Firstly, I dont think there is a way to natively do this in a simple way.

But two suggestions:

1) If you are looking just to close alerts, a Notification policy could accomplish this.  Simply match on tags and then auto-close after some time

2) The more complicated way would be using an OEC.  This could be triggered by some subset of alerts, and then a script could be created to do any functionality you want via API