On-Call Best Practices

[#1]

When being on-call, it’s important that users are receiving notifications from alerts during the time of an issue. To make sure that the on-call engineer is properly receiving these notifications, the best practice would be to do a test run before on-call hours.

Here are a few tips when testing out notifications for on-call preparation:

The first step to checking if an on-call engineer's notifications are working would be to take a look at their team’s routing rules. The routing rule can route an alert to a team’s escalation or schedule, so going over these settings to see exactly where alerts are routed to will help confirm that this is configured correctly.

A common feature that users can overlook is an escalation policy’s notification time. It would be best to check that the escalation has the accurate trigger time to the on-call user.

Schedules should be reviewed to ensure the correct on-call engineer is added and the wrong user is not woken up in the middle of the night!

The team’s routing rule, escalation, and schedule can be configured reliably, but the user's profile notification settings may not be configured. A user can check that contact methods are enabled in their profile settings. From there, the user should also check that notification rules are set to send the preferred notifications(i.e., sms, email, voice).

After a user has gone through double checking these configurations, they can use the “Create Alert” button on the alert dashboard to test their on call scenario. Just choose an API integration, add a message, and input any other details needed for the alert.