Not receiving alerts for Medium or High tickets from OpsGenie for night call rotation

[Susana Woodall]

Our team is Not receiving alerts for Medium or High tickets from OpsGenie during our night call rotation. The alerts come through just fine during the specified day call rotation M-F 8a -5p. But not during the night call time frame Sunday - Saturday 5p - 7:59a. How can this be rectified? 

[Nick H · Accepted solution]

Hi @Susana Woodall ,

I'm not entirely sure what the problem might be because I think there's a few missing pieces to this issue.

Are Medium or High alerts not being created at all during after hours?

Should users be notified of Medium or High alerts during after hours / during time frame Sunday - Saturday 5p - 7:59am? And not be notified for lower priority alerts during after hours?

If users are being notified normally during business hours, and not during after hours - while Medium / High alerts are still being created - then it's most likely a configuration that's restricting these users from being notified.

It might make sense to share a screenshot of the team's On-call tab that is receiving these alerts. Then we might be able to determine why users are/not being notified.

[Susana Woodall]

The Medium and High alert tickets ARE being created, but the notifications that they are created, are not being sent to the users that are assigned to the night call rotation for that time frame. Our current night call schedule starts Tuesday 5p and ends the following Tuesday at 5p (exception: M-F 8a - 5p is designated for Day Call and only those assigned to Day call schedule get those alerts, until the night call user's time starts). Users on Night call should be notified of MEDIUM and HIGH tickets only within that time frame specified. 

[Nick H]

Hi @Susana Woodall ,

Thanks for sharing the clarification and those screenshots! Below I'll explain the workflow when an alert is sent to a team, what I believe is happening based on your screenshots, and a configuration that might help with your use case.

When an alert is sent to a team, it flows through what's configured under the On-call tab from top >> bottom; first reviewing the team's routing rules, escalations, and on-call schedules.

Best practice is routing alerts to an escalation because escalations can be configured to notify the on-call user(s) of a schedule, then others if no action is taken on the alert. 

Your team's current On-call tab is routing alerts directly to the same schedule through both the configured routing rule, and the catch-all ELSE routing rule - essentially skipping over any escalation entirely:

A few things to note;

1 - Escalations cannot route to a schedule's rotation. Only to the on-call users of a schedule. So if multiple rotations have users on-call, they'll all be notified.

2 - If you were to select the escalation in one of your routing rules, your escalation is currently configured to notify the on-call users 10 minutes AFTER an alert is sent to the team:

If you decide to route alerts to an escalation, you may want to consider changing that first rule from 10m >> 0m so users are notified immediately when an alert flows through this team/routing rule/escalation.

As for a use case like you described, you may want to break the Day and Night rotations into their own schedule. Again, if multiple users are on-call in a schedule, they'd all be notified. Below is a pretty basic use case for something you described:

Above this team's top routing rule sends alerts created M-F 8am-8pm to a business hours escalation. This escalation will notify the users of the business hours schedule.

Alerts created outside of those days/times hit the ELSE routing rule, and route to an after hours escalation - which only notifies the on-call users of the after hours schedule.

Hope that helps! Let me know if you have any other questions.