Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,297,926
Community Members
 
Community Events
165
Community Groups

How to handle concurrent incidents with automatic incident rules defined

I have stumbled into an issue where my team (incident management) receives an alert for an incident from team A and while managing this incident team B also needs to start an incident so they send an alert to our team as well. Team B alert get's automatically associated to team A's incident.

We rely heavily on slack so this is where most alerts originate from. We tried to make the alerts unique by adding the message as a tag but that doesn't seem to work.

 

We are open to suggestions.

 

Thanks in advance!

1 answer

1 accepted

2 votes
Answer accepted

Hi Coltin, 

 

Justin here, from Opsgenie support - happy to help!

 

If I'm understanding correctly, it sounds like you have multiple groups channeling alerts to a single team within Opsgenie ('Incident Management'), and that Incident Management team is what contains the rules to spin up new incidents - is that correct?

 

If so - that may be problematic, as Opsgenie is set up to separate these incident rules out by team. Each team defines its own incident rules, and those rules are only evaluated for alerts assigned to that same team. 

 

So, one potential solution would be to set up team A and team B within Opsgenie, have them each define their own incident rules, and then route each team's alerts to their team in Opsgenie only, resulting in individual incidents for each team.

 

The other option would be to leave things as they are, and set up some additional filtering on the incident rule/s defined in the 'Incident Management' team. There's not currently a way to dynamically create new incidents, if, for example, there's a unique tag on the triggering alert, but you could add some data regarding the source of the alert (perhaps the team name), and then filter based on that value in the incident rule. 

 

There are potentially other options, but these are the two that jump out as the easiest to implement. However, if you have other questions about this, please feel free to reply here, or start up a new support ticket/chat with us - we'd be happy to help!

 

Best, 

 

Justin S.

Atlassian Cloud Support

@Coltin Rose How did you guys end up resolving this issue? I am running into a similar issue as well.

@Sterling Houghton We opted to move from all the incident rules being defined in our team to having them defined for each team in OpsGenie. We then updated our internal docs to include adding our team as a responder if in fact the incident that was created is a true incident. This helped with the transition while the teams were adjusting their alerting so false incidents weren’t reported and prevented alert fatigue with my team getting paged for non incidents.

hope this helps!

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
Community showcase
Published in Confluence

An update on Confluence Cloud customer feedback – June 2022

Hi everyone, We’re always looking at how to improve Confluence and customer feedback plays an important role in making sure we're investing in the areas that will bring the most value to the most c...

103 views 0 1
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you