Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,459,883
Community Members
 
Community Events
176
Community Groups

Can one alert be used to close other alerts ?

I am using Kapacitor to generate alerts based on thresholds.  Usually Kapacitor will call the API to close an alert when the threshold is no longer breached, alerts are closed by the {{alias}} field.

There are occasions when Kapacitor will not close the alert, namely when it stops receiving data.  In these occasions I would like to generate a new alert that will close any open alerts that meet a criteria, say matching tags.

Is this possible ?

Thanks in advance.

2 answers

I have solved this using the following method.  In my monitoring script in Kapacitor I generate two alerts based on different events, but each alert is given the same identifying key, this means that in opsgenie they get the same alias so alert deduplication works.

The first alert is generated if my threshold is breached.  so if cpu>80.

the second alert is generated when we are receiving data so we get a warning the system is working alert.  This seems counter intuitive, but I set a tag on the alert IGNORE=true.  I have an ignore filter in my opsgenie integration that looks at this tag and it isn't processed.

When we stop receiving data the second alert flips to the OK state and a alert close event is generated, this does not have the IGNORE tag set and so opsgenie closes all alerts with the same alias, this includes the generated thresholds.

It is used as we are monitoring servers in a scale set in the cloud, so these servers can and do disappear at random.

Firstly, I dont think there is a way to natively do this in a simple way.

But two suggestions:

1) If you are looking just to close alerts, a Notification policy could accomplish this.  Simply match on tags and then auto-close after some time

2) The more complicated way would be using an OEC.  This could be triggered by some subset of alerts, and then a script could be created to do any functionality you want via API

Thanks for the suggestions

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events