How to handle concurrent incidents with automatic incident rules defined

Coltin Rose

I have stumbled into an issue where my team (incident management) receives an alert for an incident from team A and while managing this incident team B also needs to start an incident so they send an alert to our team as well. Team B alert get's automatically associated to team A's incident.

We rely heavily on slack so this is where most alerts originate from. We tried to make the alerts unique by adding the message as a tag but that doesn't seem to work.

 

We are open to suggestions.

 

Thanks in advance!

Justin Sitarz

Hi Coltin, 

 

Justin here, from Opsgenie support - happy to help!

 

If I'm understanding correctly, it sounds like you have multiple groups channeling alerts to a single team within Opsgenie ('Incident Management'), and that Incident Management team is what contains the rules to spin up new incidents - is that correct?

 

If so - that may be problematic, as Opsgenie is set up to separate these incident rules out by team. Each team defines its own incident rules, and those rules are only evaluated for alerts assigned to that same team. 

 

So, one potential solution would be to set up team A and team B within Opsgenie, have them each define their own incident rules, and then route each team's alerts to their team in Opsgenie only, resulting in individual incidents for each team.

 

The other option would be to leave things as they are, and set up some additional filtering on the incident rule/s defined in the 'Incident Management' team. There's not currently a way to dynamically create new incidents, if, for example, there's a unique tag on the triggering alert, but you could add some data regarding the source of the alert (perhaps the team name), and then filter based on that value in the incident rule. 

 

There are potentially other options, but these are the two that jump out as the easiest to implement. However, if you have other questions about this, please feel free to reply here, or start up a new support ticket/chat with us - we'd be happy to help!

 

Best, 

 

Justin S.

Atlassian Cloud Support