Datadog integration and routing, best practice advice

[Chad Charowhas]

I'm working with a Datadog integration (primarily for synthetic monitoring).  It is set up and working fine, but something doesn't feel right about the limitations of how we currently have it set up and after digging around in docs and videos, I thought it was time to ask for advice.
The alerts from this integration need to be routed to very different teams according to various tags on the alerts and priorities on them.  Each group/individuals need different on-call assignments, escalation, etc. etc..
Currently, we have a single DD integration set up.  My gut is telling me this might be wrong and what we really need is a bunch of almost identical DD integrations with filters and different teams assigned.  Is this supported?  I'm cautious about trying this without confirmation as I don't want to break the existing setup, which uses a single Datadog integration with our mon team(was it wrong to assign it to a team at all?), and multiple slack integrations for notifying individual teams that pull from the DD integration.  This seems to cause us to end-run around a fair number of the team based features though, which we need.
So can I add multiple Datadog integrations with specific filters on Datadog tags, assigned to different teams.
Is there a different best practice for managing an alert source serving many different teams of mixed disposition.  Again, I need to subdivide and route the generated alerts to many different teams. 
Thanks, CC

[Nick H · Accepted solution]

Hi @Chad Charowhas ,

The configuration you mentioned can be configured under one integration - but you would need access to the Advanced tab of the integration which is only available on the Standard and Enterprise plans (or the JSM/O Premium and Enterprise plans). Note the integration needs to be global - meaning it is not assigned to any team:

Here you could configure multiple create alert actions. Each action could filter on the DD tags, priorities, etc. and assign the corresponding Opsgenie responders / teams. For example:

These actions are processed in an order of operations from top >> down, and Opsgenie can use only one at most. Order sometimes matters because Opsgenie will use the first matching action / filter without reviewing any others.

[Chad Charowhas]

That looks most excellent.  Thank you for the answer and I can see that it will work for us.

However, I'd still like to know, is the multiple integrations with filters each assigned to a different team also viable, or is this the one proper way?
I don't have this particular use case in mind, but the multi-integration model would be more applicable if multiple teams all need to see a particular alert separately, as opposed to an escalation type handoff.

[Nick H]

Hi @Chad Charowhas ,

The suggestion provided above is probably the best option. The responder field within each create alert action will only be available if the integration is global.

When an integration is assigned to a team, all alerts will send to that team. You'd need to have some workflow in place to alert other teams, or allow them have visibility into other teams' alerts.

Account Admins and Owners will have visibility into all alerts. Account Users will only have visibility into the alerts they are team members of, or if they are direct of an alert.

[Brenard J. Hawkins]

When integrating Datadog and setting up routing, it's essential to establish clear objectives and a structured plan. Begin by defining the key metrics and alerts critical for monitoring your systems effectively. Utilize tagging to categorize data and facilitate efficient routing within Datadog. Establish standardized alerting policies to ensure consistency and define thresholds for different types of alerts. Implement intelligent routing based on severity and impact, considering automation where possible to streamline processes. Regularly review and optimize your setup to stay aligned with evolving monitoring needs, while also monitoring for any performance impacts.