Can I alert multiple team integrations from a single Datadog alert?

[Greg Scott]

Hello,

I'd like certain alerts created in datadog to page multiple teams at the same time (and add them both as responders). I have a separate datadog integration for each team, and I've tried adding multiple mentions in the datadog alert (e.g. @OpsGenie-Incident_Sheriff @OpsGenie-CTW_Team). When the alert fires, only one of the teams is added as a responder (seemingly at random). Anything I can change the remedy this?

Thanks!

[Nick H]

Hi @Greg Scott ,

That is possible - but what typically happens with this use case and Datadog specifically is the alert deduplicates.

Deduplication occurs when the alias of an alert matches the alias of an open existing alert. In this case, it's happening almost immediately at alert creation for all these - and it's important to note that it can occur across multiple integrations. Again, only the alias needs to match.

You can edit what data parses in the alias field under the integration's Advanced tab. We see most customers adding a string of hardcoded data such as the integration's name to differentiate the alias for each alert:

Note as well that if you edit what parses in the alias field of the Create Alert action, you'll want to consider editing this in all other integration actions so Opsgenie can identify which alert to also close, ack, etc. through the integrations. Whatever parses in the alias field should be consistent throughout the integration:

Lastly, the Advanced tab is only available on the Opsgenie Standard and Enterprise plans (and JSM-OG Premium and Enterprise plans). If you do not see this tab, then you plan does not have this functionality - and unfortunately there isn't a workaround.

[Jeffrey Tierney]

Hi @Nick H ... thank you for this useful post! It helped with exactly the issue I was trying to solve.

Unfortunately, it seems like even though I have updated the alias on both the create and close events in the opsgenie integrations, although it successfully creates alerts on both, when sending a recovery alert from datadog, it is still only closing the alert in one of the opsgenie integrations.

If I remove the one of the responders from datadog, each opsgenie integration works as expected (opening and closing the opsgenie alerts on the messages from datadog), but when both responding integrations are present in datadog, it is creating both alerts in opsgenie, but only closing one on resolution.

Any advice you might have would be helpful, and if you need anything additional from me, like screenshots, please let me know!

Thanks!

[Nick H]

Hi @Jeffrey Tierney ,

I would first check the Logs tab to troubleshoot whether or not Opsgenie is in fact receiving both requests. 

If the request is received, there could be a log that did not match the close action's filter, match on the alias, etc. Those logs might consist of:

Skipping incomingData, no matching actions found

Delaying incomingData, alert with alias does not exist yet

or something along those lines. This would mean there might be a slight misconfiguration in the integration, the alert is already closed, etc.

If the request is not in the logs, then it's most likely a slight misconfiguration on the Datadog or other system's side - since it's not being received by Opsgenie.

[Jeffrey Tierney]

@Nick H Thank you for the quick response! 

I might just be missing it, but unfortunately I don't see the logs tab. Where should I look for that... should it be right on the integration detail page within opsgenie? And would I need to have an admin role to see it?

Thanks again!

Jeff

[Nick H]

Hi @Jeffrey Tierney ,

The Logs tab can be found under the Settings >> Debug section. If you do not see this tab under the Settings, then it's most likely a restriction with your role and permissions. 

By default, only account Admins and Owners have access to the Logs tab. If you have access to this tab - you can search by strings, adjust the timestamp, etc.

[Jeffrey Tierney]

@Nick H Thanks again! I will work with one of the opsgenie admins here to get the information.

[Shashwat Khare]

Hi @Jeffrey Tierney ,
Thanks for letting us know, the Logs should help us understand if the request is being received or not. 

[Oleksandr Shmyrko]

We have the same issue. We can create multiple Opsgenie alerts using one Datadog monitor that has multiple team integrations specified, but only one Opsgenie alert is closed when the monitor sends recovery alerts (seems it sends only one recovery alert).

Datadog support confirmed that the issue is on their side:

I have confirmation from the escalation team that the Opsgenie integration does not currently support utilizing multiple notification (@ handles) in 1 monitor. That is why the additional 2 are not coming through.

They suggested opening a feature request to support such behavior in Opsgenie integration, however this looks more like a bug.

[Shashwat Khare]

Hello @Oleksandr Shmyrko ,

Thanks for bringing this to our attention.

May I please ask you to log an email ticket with us so that we can investigate if there's an alias mapping configured on your Datadog integration and further understand why the recovery alarm from Datadog is not closing the expected opsgenie alert.

Please use this link to log an email ticket with us mentioning the integration name as well as one sample alert that should be auto-closed via the Datadog integration:

Best,
Shashwat