Triaging Data Dog integration monitors

Question

I recently integrated DataDog with OpsGenie to receive alerts when a monitor is triggered in the DataDog environment. The integration itself works perfectly, but I am having trouble with translating the levels of priority into the OpsGenie environment. For example, I might have a monitor for AWS server tracking CPU utilization where I have 2 thresholds, one for warning and one for alert. Currently, I am only able to receive alerts with the default "P3" priority level. Is ops genie able to differentiate between a warning and alert trigger from DataDog and assign different P1, P2, P3, P4, P5 priority levels?

Doug Samson · Answer

This answer covers where to put the rules in place, but I'd note that the pictured mapping won't work for Datadog in particular. (This may have changed since you wrote it!) For some reason, Datadog's priority json field, which OpsGenie picks up as {{priority}}, is not actually the P1-P5 value but an unrelated value of "normal" or "low".

The real priority doesn't have its own field, so getting it requires using {{message_title.substring(1,3)}}. The upside of this is that you can define a single Custom Value process instead of having to check for each word in a different mapping.

Doug Samson · Answer

As of March 2021, Robert's answer is correct in flow, but the specific fix suggested by the picture doesn't work.

Datadog's {{priority}} field is not the P1-P5 metric but a separate field with values {"normal", "low"}. The PN field is exposed as $ALERT_PRIORITY in webhooks, but is not a named field at all in the Datadog - OpsGenie integration. (You can see accurate samples of the DD Json output here: https://docs.opsgenie.com/docs/datadog-integration.) Instead, Datadog puts the PN value at the start of the message title (e.g. "[P4] [Warn] test alert".

The good news is that this frees you from needing a flow per priority. Instead conditioning on priority, you should use a single flow with Priority set to "Custom Value to Opsgenie Priority" as described here: https://docs.opsgenie.com/docs/priority-field. Then, set the Mapped Priority Value to {{message_title.substring(1,3)}}.

Depending on what you want to do, "Warn" vs "Alert" can be handled the same way - they're the next bracketed entry in the title. If you specifically want to alter OpsGenie priority based on that state, you could define multiple flows matching different message_titles. I don't see a way to do computations like "if warn, then priority - 1" here, but you could map e.g. P1 & P2 warns to P3 priority.

However, I'd suggest leaving priority as-is, to mirror Datadog's configuration, and then using custom alerting & notification policies for your desired behavior. For example, you can set warnings to use a slower escalation policy than alerts, or delay low-priority warnings until on-hours.

Robert · Answer

Hi @Kevin Tsai !

For this, we'll utilize the advanced menu of the Datadog integration. Since Datadog uses a different priority scheme than Opsgenie, we need to do this mapping manually using a setup like this:

Once you set up the mapping for the first Datadog priority, you can click the cog icon to create a duplicate action. This should make setting up the remaining mappings a lot easier too.

Here's a link to the advanced integration settings documentation in case you want some more background: https://docs.opsgenie.com/docs/advanced-integration-settings

Hope this helps!