How to manage duplicate alerts against jira incident

[Gabriel Zanetti]

Hello team,

My company recently bought this Atlassian product but we are not sure if we can solve and issue that we are experiencing with our actual tool.

A cluster is sending alerts to our monitoring system, sometimes from one node and some other times from another. We collect the alert and with another tool we automatically generate a ticket into Jira system.

Our system is generating tickets to Jira every time that an alert arrives (and remain there for a period of time), including something duplicate alerts. Take into consideration that the alerts can arrive from multiples nodes, "despite is the same alert."

I would like to know if Opsgenie covers that kind of situation and if I would be able to generate only the ticket that the appropriate team needs, avoiding duplicate ones. 
Again...the alert can come from 2 different sources but refer to the same issue.

Hope the situation is well explained. looking forward to your response.
Thanks a lot for your support.

[Samir · Accepted solution]

Hi @Gabriel Zanetti !

Opsgenie has a concept called alert de-duplication that is used to avoid creating multiple alerts for the same issue.

Alert de-duplication occurs based on the alias field in Opsgenie. So if an alert is created with alias = "HostA", and another alert is created with that same alias, "HostA" (regardless of which integration/source it's created from), while the 1st alert is still open, the alert will de-duplicate, and increase the count of the 1st alert instead of creating a new one.

In your integrations that the alerts are created from, you can define what the alias of alerts created through that integration will be.

e.g. in this example I have an email integration configured to create alerts with the alias = a substring of the subject of the email (alias = substring between "ID" and "Status" in the subject of the email)

So if an email is sent to this integration with subject = "Server ID XYZ status down", the alert will be created with alias = "XYZ". 

This same concept applies for all integrations.

So alert de-duplication is how you can avoid having multiple alerts created for the same issue in Opsgenie.

Hope this helps!

Thanks,

Samir

[Michael Vidallon]

Hello Gabriel,

I understand how your team wanted to minimize false alerts and alert fatigue but at the same time not miss out on important alerts.

In addition John and Samir's response, wanted to share my thoughts.

If your setup is like this?

Your monitoring system > alert collection tool > Jira > OpsGenie.

Then, you may manage those alerts from your alert collection tool or in Jira (through addons like smarthandler or similar addons) before it gets to OpsGenie.

In Jira, you would need parsing or filtering of messages to a certain keyword(s) to minimize duplicate tickets which will translate to duplicate alerts. Those keywords would identify if they need OpsGenie alerts or not. (e.g. if message has "..Critical.." then set to issue priority to critical then trigger an OpsGenie alert through webhook automation)

In your OpsGenie JSM integration setup, you can manage or limit duplicate alerts (de-duplication) of the same jira ticket and jira event (like issue_created) by using {{alias}} in the Alias field under Create Alert > 'Alert Fields'. (similar to what Samir is pointing at in his response).

On the other hand, if this applies to your team, you may want to consider looking at OpsGenie Heartbeat (https://support.atlassian.com/opsgenie/docs/add-heartbeats-to-monitor-external-systems/) . This setup would look like this:

You'd need to setup in OpsGenie an API integration that you would be consumed in your monitoring tool and sends heartbeat to OpsGenie. Depending on your requirement, you can setup a trigger to either create a jira ticket out of that alert or heartbeat or some custom actions. You would need to setup one heartbeat per server your monitor.

Setup varies if your JSM is on cloud or on-prem (server / data center).

Hope this helps.

[Gabriel Zanetti]

Hi Michael,

Thanks for your thoughts. Much appreciated.
No exactly. I'm planning to use Osgenie to centralize all the alerts that come from 5 different monitoring systems and then automatically generate a jira ticket. So I'm looking to simplified the infra.
Something like this

(monitoring tool) > Opsgenie > Jira

[Michael Vidallon]

Great, I believe this might be helpful then. This is found in the Advanced tab of the Jira Integration setup (OpsGenie to Jira integration).

All the best!

[John M]

Hi Gabriel,

If you can configure the alias field to match on alerts from different sources which still refer to the same issue, that would cause the alerts to be deduplicated. 

Deduplication is when, instead of creating a new alert, the alert count of another open alert with a matching alias is increased. 

You can read more about it here: https://support.atlassian.com/opsgenie/docs/what-is-alert-de-duplication

[Gabriel Zanetti]

Thanks John!