How to alert entire team + escalate to on-call engineer?

[Deleted user]

Hi everyone,

We have a team of four engineers and I would like to set up alerting as below

-All engineers get SMS'd immediately

-If not ack'd after X minutes, call on-call engineer

-If not ac'd after a further X minutes, call specific person.

How would I achieve this? Thanks

[Emirhan]

Hello @[deleted], 

Thank you for contacting Opsgenie Community!

Within the escalation policies, you'll be able to configure which user/schedule/team should be notified if the alert is not acknowledged/closed within a defined period. Here is how your escalation policy should look:

The notification methods of how users receive notifications can be configured under the profile settings of each user. On the other hand, we do have a feature called Central Notification templates that provides the ability to define shared notification settings that are managed from a single place and can be assigned to user roles.

https://docs.opsgenie.com/docs/central-notification-templates

That being said, we do not generally recommend our customers to use SMS & Voice notifications at the very beginning of the notification flow while you have an email and mobile app notifications options. Mostly, the recommendation is using the SMS & Voice notifications as a back-up plan where your on-call users are not responding to the alerts, like 30 minutes later, after the alert created so you can have more aggressive notification flow if the expected actions are not taken. 

I hope my answer helps.

Please, feel free to let me know if you have anything else in mind, I would be happy to assist!

[Deleted user]

Hi @Emirhan ,

Thanks for the reply. Our use case relies on SMS/Voice as we need to react very quickly to our issues.

To simplify, what would be the escalation/notification settings for this? I can't work out how to configure it like this despite playing around for a few hours.

Engineer A is on-call, Engineer B is manager

---- SMS Engineer A, B & C immediately

---- if not ack'd in 5 minutes, call on-call Engineer (A)

---- if not ack'd in 10 minutes, call manager (Engineer B)

Thanks,

Andy

[Deleted user]

Hey @Emirhan - do you have any update? Thanks

[Deleted user]

@Emirhan ?

[Ryan Howlett]

I assume you never had a reply on this one?

I have a similar scenario whereby we would like to notify different people with different notification methods.

i.e. Email goes out for every alert to all users in the group.  However, at the same time, we want sms/app alerts going to the on-call engineer for out of hours support.  At that point, we would then want escalation processes to run which would then sms/app alert to all members of the team.

[Deleted user]

I've posted a support ticket, hopefully they will respond here too :-)

[Connor Eyles]

Hi All, I have recently worked with @[deleted] on this one and thought I would share my findings on this one.

Looking at this logic here --------

Specifically, we have a time of five engineers - A,B,C,D,E. One of them is the manager (B) and any one of them (A,B,C,D,E) could be on-call at any time.

When we have an issue, we would like the following

• All engineers (A,B,C,D,E) receive a SMS.
• If not ack'd in 5 minutes, only the on-call engineer is called.
• If not ack'd in 10 minutes, it is then escalated to manager and they are called.

-------------

I've gone over your request and this particular type of logic kind of takes away the logic of how Opsgenie is configured. In your first statement, you mention you want to have everyone notified by SMS, In Opsgenie once someone is alerted they are considered a responder, Have a read of this document here https://docs.opsgenie.com/docs/alert-recipients-and-teams

If we were to alert all users at the start with an SMS they would be considered a responder and they should action said alert but by the sounds of it in your case what you want to achieve is to let the whole team know about the alert via SMS "All engineers (A,B,C,D,E) receive an SMS." rather then actually have them action they are alerted, only the on-call engineer should action. If not acknowledged then escalate up.

So right now this current logic will not work, unfortunately.

I do have a possible workaround which I believe could work better for you, Do you and your team use Slack? (https://slack.com/intl/en-au/) or Microsoft Teams? I believe this could be very useful for your scenario so you can differentiate between simply being made aware of an alert and actually needing to action the alert. 

So what you could do is:

• All engineers (A,B,C,D,E) receive a Slack notification as the alert will be posted in the team channel.
• Alert on-engineer
• If not ack'd in 10 minutes, it is then escalated to a manager and they are called.

The Engineers have a notification policy to be SMS'd instantly when alerted and then called after 5 mins.

The Manager will have a policy to be called instantly once escalated.

This way when they receive a slack notification on their phones they know this is a "hey you need to know about the alert but someone else is looking after it" but if they receive an SMS or a call its a "Hey you need to action this"

With the slack integration (https://docs.opsgenie.com/docs/slack-app-integration) you can also perform actions like acknowledge the alert.