Opsgenie/JSM Alerts deduplication

[Phil Beckman]

We are users of JSM premium which bundles the product formerly known as Opsgenie into the mix. Things are working pretty well with one exception that we cannot figure out.

The problem we have is around alert deduplication. I'm going to simplify the problem a bit just for the sake of berevity.

Assume we run a number of services on a single virtual machine. If that machine gets overwhelmed (or just goes offline) we will then get 3 or 4 alerts from our monitoring software (UptimeRobot in this case) all within a few minutes of each other telling us "Service XYZ has stopped responding". We really only need one alert and the rest would be considered to be duplicates. In short, if we get more than one alert from a given set of services all within say 5 minutes of each other, we'd like to consider all but the first one to be duplicates.

All the documentation indicates that alerts will be deduplicated if the alias on an incoming alert matches the alias on an already existing alert. That sounds like exactly what we'd want except there does not appear to be any way to set the alias on an incoming alert based on some conditions. What I can't find is anything that describes how we can get hooked into the alert creation process in order to assign an existing alias to alerts 2-n in the scenario above.

We're quite capable of writing scripts, hooking into event models or anything else out there but I can't find anything that would indicate that there's any way to get out proverbial fingers in there in the first place.

Am I missing something?

[Nicola Carraro]

I agree with @Phil Beckman  that information on how to use Aliases is lacking at best. 

I have similar scenarios and here's how I'm managing:

1. Use a policy to match the incoming alert
2. Set the Alias via the policy

Of course, for this to work you have to have received the alert at least once, so that you can create the policy, but it'll work for every similar alert after that.

Hope this helps.

[Phil Beckman]

That's actually an interesting idea but I don't think it will work for us. The problem is that we would not always want to set the alias to a new value. 

If service XYZ goes down because it honestly crashed then we would want that alert to come through. On the other hand, if service XYZ, service abc and service def all go down within a minute of each other then likely 2 of those 3 should be deduplicated.

The best answer I have seen thus far involves writing something like a Lambda function in AWS where the Lambda function if what the alerting software (UptimeRobot) notifies and then the Lambda function has intelligence enough to decide what to send on to JSM and what not to.

It feels like OpsGenie combined with JSM is almost but not quite exactly what we need.

[Nicola Carraro]

I agree that in this case it would not work. For those scenarios we use monitoring systems that understand the dependencies between platforms/services and decide what alert to trigger based on the condition.

[Bruna Silva]

Hello @Phil Beckman ,

Thank you for reaching out to the Atlassian Community!

Alert deduplication is a primary feature of JSM Operations and there are currently no available options to disable or alter this standard functionality. If the below conditions are matched then any newly received alerts will not be created, and the existing alert will be deduplicated. This behavior is by design, and it is there to reduce alert fatigue. This is because there could be multiple incoming alerts from other tools/sources about the same service impact.

Criteria:

• A new alert request has been received with an existing alias.
• The existing alert is still open.
  • A new alert notification is sent if the dedup count reaches 100.

The alias is a set value on the JSM Operations side, but you can configure it to be a specific value from the information JSM Operations is receiving from your integration. With that in mind, if you can set this conditional logic in your monitoring software for a value that is specified to map to the alias, then you would be able to achieve a similar result you're looking for.

More details about this case can be found in the following document:

• Alert fields

If you have any other questions regarding this topic, please let us know.