Aggregate multiple alerts from different teams to one incident?

Question

Hi all,

we'd like to solve following use case with Opsgenie:

Team A owns an internal service A, which is called by teams B, C and D within their external services B, C and D. Every team monitors its own service and sends an alert when something seems to be faulty.

Now we want to implement a ruleset, which creates an incident when all four teams are receiving alerts at the same time, because that indicates that there's an issue with service A which definitely affects all service-users. This incident shall be owned by team A, whereas teams B, C and D shall be stakeholders.

As the Incident Rules are to be configured on team level, they seem only to apply for those alerts in which the team is already being alerted as responder. But that doesn't fit for us, as team A is not necessarily responder for team Bs alerts (and vice versa, of course).

So, how can we realize that? Is there something like cross-team incident rules? Or any workaround you could think of?

Thanks in advance for your ideas!

Best regards,

Stefan

Stefan Draber_kwaltd · Answer

Hi @Darryl Lee ,

thanks for your feedback, unfortunately it doesn't solve my problem yet, as Team A's incident rule still is only triggered by Team A's alerts.

Our use case is: when there's an alert for Team A and another alert for Team B at the same time, then an incident should be created.

When there's only one of these alerts, then it should only be alerted to the affected team (alert A to Team A, alert B to Team B) .

Let me try to give you a scenario: Imagine a backend service for calculation and a frontend service providing a calculator to the user.

When there's an alert for calculation-service but none for the frontend, then it seems to be an internal issue in the calculation-service which doesn't affect the frontend and therefore only has to alert the calculation-team.

When there's an alert for the frontend but none for the backend, then it seems to be an issue with the frontend which doesn't affect the calculation-service and therefore only has to alert the frontend-team.

Only when these two alerts come together at the same time we can be sure that there's an issue in the calculation-service, which also affects the frontend and thus our users. So we want to handle it as an incident owned by the calculation-team, which regulary informs its stakeholders via incident updates and so on.

I hope this sheds some light on what we want to achieve :)

Best regards,

Stefan

Darryl Lee_2jimne · Answer

Hi @Stefan Draber ,

This is Darryl. I am here to help.

If you would like to leverage the team's incident rule to create Opsgenie Incident for an Opsgenie Service that is owned by 1 team while expecting the Incident Responders List to include the other teams, you may configure the Responders > Add responder teams on that Opsgenie Service page to contain the wanted teams.

For example:

Then, on your team's Incident Rule, fill in to point to this Service you just updated. This will make the created Opsgenie Incidents inherit the Responders List from that Service.

For example:

Hope this helps.

Kind regards, 
Darryl Lee
Support Engineer, Atlassian