3 seemingly obvious but helpful Opsgenie alerting tips

[#1]

This list was inspired by some recent chats that I've had with newer Opsgenie customers. Sometimes people get stuck doing something a certain way, or are forcing solutions to work together because of legacy, lack of budget, or "this is the way it's always been done" etc. When that happens, sometimes the features we use in a streamlined tool, aren't as intuitive as we may think. If you're a seasoned Opsgenie user the below might seem obvious. But if you're new to Opsgenie and/or incident management tooling in general, these little tidbits can save you some time and future headaches. 

1. Only acknowledge an alert if you plan on doing something about it. 

The purpose of an alert is to notify the right people so that they can take action and solve or prevent a problem. If you acknowledge an alert (possibly as a reflex to receiving the notification), but don't plan to do anything about it, you're sending a confusing message. Your team and other responders will assume the problem is being taken care of, and then the alert becomes the tree that fell in the empty woods (no one heard it, no one knew about it). 

2. You do have to acknowledge an alert to prevent notifications from continuing. 

Each team has a list of escalation policies, routing rules, and on-call schedules. These all work together to notify the right people on call so they can take action quickly. Escalation policies work to prevent alerts from sliding under the radar and will continue to escalate the alert until someone acknowledges it. If a responder does not acknowledge an alert but begins working on it, the escalation policy will continue to notify folks until acknowledgement, and create confusion and likely duplicate work on the team. 

3. "Seeing"  an alert, stops your notification flow. 

Opsgenie is smart, and helps to reduce and prevent alert fatigue. If you're logged in to Opsgenie when an alert comes in (whether it be open on your phone or computer) and you click on that alert to see more information you will not receive notifications for that alert. Why? Because Opsgenie knows you "saw" it, and thus there is no need to notify you again. 

That's all for today. If you have a helpful tip or trick feel free to share it in the comments! 

[Jimmy Seddon]

This is great @Kate Clavet!  I'm just getting started with help of other awesome members of your team so this is really helpful advice!

[Kate Clavet]

That's awesome @Jimmy Seddon ! Thanks so much for sharing that with me. If you're willing I'd love to set up a chat to hear more about your specific experience. 

Best,
Kate 

[Andrew Laden]

While this is generally correct, it isn't absolute.

1: There are options to re-notify after a period of time.

2: While the default is to stop escalation on acknowledgement, you can chose to set it otherwise.

One thing that has bit us before. If a user is in multiple escalation levels, then they will not get notified if the issues is escalated.

for example. a common escalation case. Start with an on call user. Then escalate to thier manager, then escalate to the whole team.

We have people who expected that the oncall user would get notified at the initial level, and then again when the issue is escalated to the whole team. This is not the case. Once a users has been notified, even if a further escalation would make it seem like they would get notified again, they will not.

Regardless of escalation rules. (and not withstanding renotification settings) Once a user has been notified, they will not get notified again. (just like if they have seen the notification, they will not get further notifications even if thier personal rules would send a later alert.

[Kate Clavet]

Thanks so much for sharing @Andrew Laden ! You are certainly correct, it's not absolute. That's what makes Opsgenie such a helpful tool for so many teams, it is truly flexible and can be created to suit various use cases. Thanks for sharing your insights, they are certainly helpful! 

[Jack Brickey]

I thought I would add a tip I just learned....

Use "No-one" in a rotation to 'skip' time periods. I was scratching my head on how to have someone on-call every-other weekend. By including "no-one" in the Participants field of a rotation you can make that happen...

Jack --> No-one --> Jack --> No-one...

[Deleted user]

#3 seems really poorly planned.  

Have you ever been actively using your phone when a notification comes up?  Maybe you are in the middle of a text, email or some other emergency task and the notification pops up right under your finger and it accidently opens the notification.  You quickly switch back to what you are doing to complete that task.  From that point on, you no longer get alerts, BUT your coworkers in the escalation path do...and you have no idea because OG has decided that you saw the alert and no longer wish to be notified.

Maybe the alert has your full attention. You read it and hit the ACK button and immediately start working to address the issue....except for whatever reason your phone didn't register the ACK.  You've seen the alert, so OG is done with you...but it's not done with the escalation path...no sir, OG is more than happy to wake them up at 3AM for an issue you are already working to resolve and have no idea they are being alerted on.

Are both of those issues my fault for not properly addressing the alert?  Absolutely.  However life happens and we don't live in a perfect world where only one thing happens at a time and everything can have your fullest attention, and it doesn't change the fact that I'm the on-call.  When an issue arises that was important enough to alert me at 3AM or in my off hours, I want my phone to blow up until I respond to it.   Start with a push notification, if I don't get that, then a text, then a phone call and repeat that until I ack the alert.  I don't want to be cut out of the alert process because OG is "smart" and thinks I don't want to be bothered any longer.  I want to be bothered until I intentionally ACK that alert!

I don't care how "smart" OG is...let me decide when I don't want to be bothered anymore.