How to alert entire team + escalate to on-call engineer?

Deleted user June 14, 2020

Hi everyone,

We have a team of four engineers and I would like to set up alerting as below

-All engineers get SMS'd immediately

-If not ack'd after X minutes, call on-call engineer

-If not ac'd after a further X minutes, call specific person.

How would I achieve this? Thanks

1 answer

0 votes
Emirhan
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 16, 2020

Hello @[deleted], 


Thank you for contacting Opsgenie Community!

Within the escalation policies, you'll be able to configure which user/schedule/team should be notified if the alert is not acknowledged/closed within a defined period. Here is how your escalation policy should look:

image (23).png

 

The notification methods of how users receive notifications can be configured under the profile settings of each user. On the other hand, we do have a feature called Central Notification templates that provides the ability to define shared notification settings that are managed from a single place and can be assigned to user roles.

https://docs.opsgenie.com/docs/central-notification-templates

 

That being said, we do not generally recommend our customers to use SMS & Voice notifications at the very beginning of the notification flow while you have an email and mobile app notifications options. Mostly, the recommendation is using the SMS & Voice notifications as a back-up plan where your on-call users are not responding to the alerts, like 30 minutes later, after the alert created so you can have more aggressive notification flow if the expected actions are not taken. 

 

I hope my answer helps.

Please, feel free to let me know if you have anything else in mind, I would be happy to assist!

Deleted user June 16, 2020

Hi @Emirhan ,

Thanks for the reply. Our use case relies on SMS/Voice as we need to react very quickly to our issues.

To simplify, what would be the escalation/notification settings for this? I can't work out how to configure it like this despite playing around for a few hours.

Engineer A is on-call, Engineer B is manager

---- SMS Engineer A, B & C immediately

---- if not ack'd in 5 minutes, call on-call Engineer (A)

---- if not ack'd in 10 minutes, call manager (Engineer B)

Thanks,

Andy

Deleted user July 13, 2020

Hey @Emirhan - do you have any update? Thanks

Deleted user August 3, 2020
Ryan Howlett August 21, 2020

I assume you never had a reply on this one?

I have a similar scenario whereby we would like to notify different people with different notification methods.

i.e. Email goes out for every alert to all users in the group.  However, at the same time, we want sms/app alerts going to the on-call engineer for out of hours support.  At that point, we would then want escalation processes to run which would then sms/app alert to all members of the team.

Deleted user August 23, 2020

I've posted a support ticket, hopefully they will respond here too :-)

Connor Eyles
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 3, 2020

Hi All, I have recently worked with @[deleted] on this one and thought I would share my findings on this one.

Looking at this logic here --------

Specifically, we have a time of five engineers - A,B,C,D,E. One of them is the manager (B) and any one of them (A,B,C,D,E) could be on-call at any time.

When we have an issue, we would like the following

  • All engineers (A,B,C,D,E) receive a SMS.
  • If not ack'd in 5 minutes, only the on-call engineer is called.
  • If not ack'd in 10 minutes, it is then escalated to manager and they are called.

-------------

I've gone over your request and this particular type of logic kind of takes away the logic of how Opsgenie is configured. In your first statement, you mention you want to have everyone notified by SMS, In Opsgenie once someone is alerted they are considered a responder, Have a read of this document here https://docs.opsgenie.com/docs/alert-recipients-and-teams

If we were to alert all users at the start with an SMS they would be considered a responder and they should action said alert but by the sounds of it in your case what you want to achieve is to let the whole team know about the alert via SMS "All engineers (A,B,C,D,E) receive an SMS." rather then actually have them action they are alerted, only the on-call engineer should action. If not acknowledged then escalate up.

So right now this current logic will not work, unfortunately.

I do have a possible workaround which I believe could work better for you, Do you and your team use Slack? (https://slack.com/intl/en-au/) or Microsoft Teams? I believe this could be very useful for your scenario so you can differentiate between simply being made aware of an alert and actually needing to action the alert. 

So what you could do is:

  • All engineers (A,B,C,D,E) receive a Slack notification as the alert will be posted in the team channel.
  • Alert on-engineer
  • If not ack'd in 10 minutes, it is then escalated to a manager and they are called.

The Engineers have a notification policy to be SMS'd instantly when alerted and then called after 5 mins.

The Manager will have a policy to be called instantly once escalated.

This way when they receive a slack notification on their phones they know this is a "hey you need to know about the alert but someone else is looking after it" but if they receive an SMS or a call its a "Hey you need to action this"

With the slack integration (https://docs.opsgenie.com/docs/slack-app-integration) you can also perform actions like acknowledge the alert.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
AUG Leaders

Atlassian Community Events