Anyone using OpsG for afterhours alerting?

[Jack Brickey]

Hi OG Community! I am looking for thoughts from those of you using OG on how it might fit my needs. First let me state that I'm not totally unfamiliar with OG and in fact played with it a bit back in 2018 briefly. However, while IT incident management likely will be of interest to me in the future my current needs are different.

High-level project goal

I have a logistics group that is responsible for shipping equipment to various locations worldwide.  We have a software application that handles all incoming request for said equipment that the team uses to fulfill these orders and maintain communications w/ the customer. This software is perfectly capable for managing our normal business hours activities however, for after hours it falls short where notifications are concerned.

What I need is a solution that can receive an email (order request) and can reliably send notifications to the on-call person. Notifications would typically look like this:

1. SMS to primary on-call team agent for which the agent needs to respond w/ say 1m. If they do not respond then...
2. Phone call to primary on-call team agent. If they do not answer then...
3. SMS to secondary on-call agent and if they do not respond in say 1m then...
4. Phone call to secondary on-call team agent. If they do not answer then...
5. Escalate L1 (SMS --> phone call). no response...
6. Escalate L2 (SMS --> phone call)

Other reqs:

• when any agent responds the notifications stop
• the ability to have the notification process continue thru multiple iterations
• the ability to define on-call agent calendar (daily primary, secondary, etc.)
• the ability to leverage a default on-call calendar
• the ability to have notifications schemes and on-call calendars based upon variables, e.g. customer, priorities, etc.
• the ability to create reports and stats on the teams response times, e.g. #alerts responded to in #1, #2, #3 (above) etc. Ideally, these reports should be accessible to any Stakeholders
• the ability to configure email notifications to any stakeholders associated w/ the different notification rules shown above. Such as, email the stakehoders for #5, #6

As you can see I'm considering OG for an on-call-notification solution not really "incident management". From my early testing, reviewing and demos it would seem it would work fine but I'm hoping maybe others have actually set something like this up and can give me their unbiased opinions or even suggest alternate solutions.

TIA!

[Jack Brickey]

@scott_boisvert, awesome input! Thank you so much. Based upon my past research I was pretty confident it would meet my primary requirements but it helps to hear from actual experienced users. I suspect I will try the free version and play w/ it to gain personal experience and decide how to proceed from there.

[scott_boisvert]

@Jack Brickey I'm using Opsgenie currently. Similar boat, we're not using the incident management yet, but using it for on-call notifications. So, Opsgenie does have an e-mail integration, we set each team up with an e-mail integration. I also wrote a Confluence macro that utilizes the Opsgenie API to pull the all the e-mails and the team name into a table on a Confluence page so all the e-mail addresses are in one place.

You're escalation is interesting, I like the idea of first notifying by text, then notifying by phone, etc.... I'm not entirely sure that's possible, but let me do a little more research in my sandbox. I know my users would love this, because they have "complained" that they get the phone, text and app push all at the same time for P1's (just how we set it up).

From my knowledge you set a notification template that indicates what contact methods are used given specific conditions on the alert, for instance a P1 alert the user would get phone call, text message and push notification to the app, where as a P4, would just get an e-mail.

The rest of the escalation is doable, i.e. notify the primary first, then the secondary and so on. We escalate every 5 minutes from the primary to the secondary, to the team's manager, then director then AVP.

As for the other requirements:

• when any agent responds the notifications stop
  • Alerts will stop when the agent ACKS the alert (this can be done from the phone call they receive, the app or the web portal).
• the ability to have the notification process continue thru multiple iterations
  • There is the ability to set alerts to loop back through the escalation policy.
• the ability to define on-call agent calendar (daily primary, secondary, etc.)
  • Each team has the ability to create multiple schedules and multiple rotations with the schedules based on a calendar.
• the ability to leverage a default on-call calendar
  • Can you explain what you mean here?
• the ability to have notifications schemes and on-call calendars based upon variables, e.g. customer, priorities, etc.
  • Yes, you can set up notifications based on variables within the alert like priority, e-mail address the alert came from, etc.
• the ability to create reports and stats on the teams response times, e.g. #alerts responded to in #1, #2, #3 (above) etc. Ideally, these reports should be accessible to any Stakeholders
  • Opsgenie has a bunch of built in reports on response times. Stakeholders, which do not require an Opsgenie license do not have access to reports (https://docs.opsgenie.com/docs/user-roles), though you can create custom roles, but that would take up a license.
• the ability to configure email notifications to any stakeholders associated w/ the different notification rules shown above. Such as, email the stakehoders for #5, #6
  • Yes, stakeholders can be configured to receive, stakeholder notifications. 

[Kate Clavet]

Hi @Jack Brickey
Email me at kclavet@atlassian.com and I'll get that set up for you! 

[Jack Brickey]

Hi @Kate Clavet , assume you are associated w/ Opsgenie? Actually, we are trialing now and I was going to try to reach out to someone at OG to see if it would be possible to spend 15m on the phone to gain insight and guidance. Previously, I have had a demo presented to my IT team but the current use case is a bit different and consequently am trying to see how best to leverage the tool. LMK, if this is something you or someone else could accommodate.

[Kate Clavet]

Just joining the conversation now, great post @Jack Brickey and thanks so much @scott_boisvert for the awesome answer! Let me know if you have any other questions while getting set up, happy to help! 

[scott_boisvert]

@Jack Brickey Confirmed you can set it up so that the notifications go out Text at 0 minutes, Phone at 5 minutes, etc... This is done in the Notification Rule of the Notification Policy.