Opsgenie/JSM Alerts deduplication

Question

We are users of JSM premium which bundles the product formerly known as Opsgenie into the mix. Things are working pretty well with one exception that we cannot figure out.

The problem we have is around alert deduplication. I'm going to simplify the problem a bit just for the sake of berevity.

Assume we run a number of services on a single virtual machine. If that machine gets overwhelmed (or just goes offline) we will then get 3 or 4 alerts from our monitoring software (UptimeRobot in this case) all within a few minutes of each other telling us "Service XYZ has stopped responding". We really only need one alert and the rest would be considered to be duplicates. In short, if we get more than one alert from a given set of services all within say 5 minutes of each other, we'd like to consider all but the first one to be duplicates.

All the documentation indicates that alerts will be deduplicated if the alias on an incoming alert matches the alias on an already existing alert. That sounds like exactly what we'd want except there does not appear to be any way to set the alias on an incoming alert based on some conditions. What I can't find is anything that describes how we can get hooked into the alert creation process in order to assign an existing alias to alerts 2-n in the scenario above.

We're quite capable of writing scripts, hooking into event models or anything else out there but I can't find anything that would indicate that there's any way to get out proverbial fingers in there in the first place.

Am I missing something?

Nicola Carraro · Answer

I agree that in this case it would not work. For those scenarios we use monitoring systems that understand the dependencies between platforms/services and decide what alert to trigger based on the condition.

Phil Beckman_sewwsd · Answer

That's actually an interesting idea but I don't think it will work for us. The problem is that we would not always want to set the alias to a new value.

If service XYZ goes down because it honestly crashed then we would want that alert to come through. On the other hand, if service XYZ, service abc and service def all go down within a minute of each other then likely 2 of those 3 should be deduplicated.

The best answer I have seen thus far involves writing something like a Lambda function in AWS where the Lambda function if what the alerting software (UptimeRobot) notifies and then the Lambda function has intelligence enough to decide what to send on to JSM and what not to.

It feels like OpsGenie combined with JSM is almost but not quite exactly what we need.

Nicola Carraro · Answer

I agree with @Phil Beckman  that information on how to use Aliases is lacking at best.

I have similar scenarios and here's how I'm managing:

* Use a policy to match the incoming alert
* Set the Alias via the policy

Of course, for this to work you have to have received the alert at least once, so that you can create the policy, but it'll work for every similar alert after that.

Hope this helps.