Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,295,646
Community Members
 
Community Events
165
Community Groups

Can one alert be used to close other alerts ?

I am using Kapacitor to generate alerts based on thresholds.  Usually Kapacitor will call the API to close an alert when the threshold is no longer breached, alerts are closed by the {{alias}} field.

There are occasions when Kapacitor will not close the alert, namely when it stops receiving data.  In these occasions I would like to generate a new alert that will close any open alerts that meet a criteria, say matching tags.

Is this possible ?

Thanks in advance.

2 answers

I have solved this using the following method.  In my monitoring script in Kapacitor I generate two alerts based on different events, but each alert is given the same identifying key, this means that in opsgenie they get the same alias so alert deduplication works.

The first alert is generated if my threshold is breached.  so if cpu>80.

the second alert is generated when we are receiving data so we get a warning the system is working alert.  This seems counter intuitive, but I set a tag on the alert IGNORE=true.  I have an ignore filter in my opsgenie integration that looks at this tag and it isn't processed.

When we stop receiving data the second alert flips to the OK state and a alert close event is generated, this does not have the IGNORE tag set and so opsgenie closes all alerts with the same alias, this includes the generated thresholds.

It is used as we are monitoring servers in a scale set in the cloud, so these servers can and do disappear at random.

Firstly, I dont think there is a way to natively do this in a simple way.

But two suggestions:

1) If you are looking just to close alerts, a Notification policy could accomplish this.  Simply match on tags and then auto-close after some time

2) The more complicated way would be using an OEC.  This could be triggered by some subset of alerts, and then a script could be created to do any functionality you want via API

Thanks for the suggestions

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Posted in Jira Service Management

Jira Service Management Documentation Opportunities

Hello everyone, Hope everyone is safe! A few months ago we posted an article sharing all the new articles and documentation that we, the AMER Jira Service Management team created. As mentioned ...

306 views 0 9
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you