We've set up notifications based on alert priorities. P1 through P3 alerts will notify dev team users, staggering times between escalations. This was agreed to by our management team and I am not inclined to change it. All alerts for any given service will also go to Slack, regardless of priority. A typical P2 alert, for example, will follow something like
- (Immediately) notify internal cross-support team
- (5 min) notify service team member on call
- (10 min) notify next service team member on call
- (15 min) notify all service team users
- (20 min) notify all service team admins
Some users are asking to be notified only for P1 alerts. My question is, without significantly modifying this structure, how can users be notified by email for non-critical alerts (P2-P5) and called for P1 alerts? There doesn't seem to be a distinction between notification mechanisms within OG so I'm not sure how to keep my users AND management happy. (FWIW, the "suck it up" response was not well received!😂)
Also, I've considered letting Slack notify for "P1" as a keyword, but it happens across ALL Slack channels a user is subscribed to. We have too many individual services to make this work effectively.
