Update: We are re-evaluating escalation limits in Opsgenie

[#1]

On Sep 7, 2022, we announced that new limits for escalation policies in Opsgenie would go into effect on Sep 21, 2022. We’ve since received feedback from many of you about the impact of the proposed changes. As a result of this:

• Our engineering team is re-evaluating how we approach escalations in Opsgenie overall and we will provide you with an update soon. 

• As part of this effort, we are reaching out to many of you to gather additional feedback about your escalation rules use cases.

• We will NOT add new limits to escalation policies on Sep 21, 2022.

We apologize for how the changes were previously communicated. Thank you for your feedback. We hear you.

If you have any additional questions, please drop them in the comments section below. Thank you for being an Atlassian customer.

+++The content below is outdated as of 9/14/22, please read above for the latest update on escalation policies in Opsgenie+++

TL;DR: We’re announcing new limits for escalation policies in Opsgenie effective on September 21, 2022. This change does not impact any escalation rules that have already been set up. The new limits are: Escalation policies per team: 7, escalation rule count per escalation policy: 10, and escalation rule repeat count: 5.

We’re announcing the following limits for escalation policies in Opsgenie, effective on September 21, 2022.

The new limits are:

• Escalation policies per team: 7

• Escalation rule count per escalation policy: 10

• Escalation rule repeat count: 5

Why are we making this change?

We’re making this change to improve the performance and reliability of Opsgenie. In the past, if a customer with a large number of escalation policies, and multiple rules and repeats per escalation experienced a flood of alerts it could degrade the performance, and reliability of Opsgenie. In some cases, this has resulted in downtime and incidents, which are unacceptable for customers to experience. We feel these new limits enable every customer to continue to use escalations while also keeping the system up and running consistently.

If you currently have more escalation policies than the new limits listed above, that’s ok, all your escalation policies will continue to work - you don’t need to do anything.

However, if you have an escalation that is greater than the new limit and you make an edit or an addition to that escalation, an in-app message will notify you that you need to adjust the escalation to meet the new limit, or exit the edit window and keep the escalation at its current configuration.

If you have any questions about these new limits, please leave a comment below. 

[Kevin Mahoney]

So you’re removing functionality from your service? After evaluating this v. PagerDuty… I settled on OpsGenie as I hadn’t tried it and it was basically a coin toss.

unfortunately my evaluation didn’t include looking at mobile app reviews

the fact that this email hit my inbox today to say opsgenie is becoming objectively worse, not better, cements the fact that I’ll never recommend it again. 

[Todd Minnella]

In the announcement above, when it says "Escalations per team" - does that refer to the total number of Escalation policies in a team?

In the example above, how is the limit of "7 escalation rules to a team" being calculated - is it the sum of all rules in the team? I see 3 policies and 6 rules above. And, if the limit is 7 escalation rules in a team, how would the "Escalation rule count of 10 within an escalation" limit be reached, if the limit is 7 rules across the entire team?

In any case, if we have a complex set of routing rules that necessitate the use of more policies and/or rules than will soon be allowed, are these limits adjustable by support?

Thank you!

[john.wussow]

Can we get an explanation as to why this change is being made? Is it technically challenging to have that many rules in an Escalation policy?

[Andrew Laden]

This is a big limitation change. We have a lot of teams that use multiple escalations. We have 5 priority levels, and each level has a daytime vs nighttime escalation rule. So that puts us at 10, not counting some other generic cases. 

I know you say that existing ones will not be affected, but that is simply not true. I cant not make any adjustments to my rules without having to cut them down. And I can no longer replicate my setup to any new teams that I create.

Not that I expect atlassian to reconsider because they never seems to reconsider any decision no matter what their customers say, but PLEASE reconsider this limitation.

It would also help if you clarify and use your own terminology. When you say 7 escalations per team, what is an escalation. Do you mean 7 escalation policies per team? If so, then say that.

 Does this also affect Routing rules?

Again, PLEASE DO NOT MAKE THIS CHANGE!

[tmccormack]

I'm curious about the reasoning as well. Was it a big performance hit?

[Michael J George]

While this change won't affect our simple deployment, I am also curious why this limit is being imposed.

[Samir]

Hi All - thanks for your feedback and questions so far on these new limits. We will aim to update the community post with more detail on why the change was made and a FAQ section to clarify the questions that have come up...

@Todd Minnella 

In the announcement above, when it says "Escalations per team" - does that refer to the total number of Escalation policies in a team?

Yes - there is a limit of 7 escalation policies per team. We just updated the post to make that more clear. Within each escalation policy, there is a limit of 10 rules/steps. So the first screeenshot in the post shows 3 escalation policies. The 1st escalation policy has 4 rules/steps, and the other 2 each have 1 rule/step.

@Andrew Laden  Thanks for sharing your use-case and feedback.

There may be a way to achieve your use-case by having 5 escalation policies (1 for each priority), and then configuring a schedule with a daytime night-time rotation. So the routing rules will route to an escalation policy based on priority, and the escalation policy will be set-up to notify on-call users in the schedule, and if it's daytime the users in the daytime rotation will be on-call, if it's night-time then the users in the night-time rotation will be on-call. And if the next steep is "notify next on-call user", it will know to choose the next on-call user in the relevant rotation (e.g. if it's daytime, next user on-call in the daytime rotation)

However I understand there may some other constraints that would prevent this approach from working for you.

There is no change to the number of routing rules you can add, just these 3 limits on escalation policies.

[Andrew Laden]

@Samir That is basically what I am doing. Which is why i need all those escalation policies

1. I have a set of routing rules that route to an escalation based on Priority. (ie, events with a priority value of "1" route to "Pri 2 Escalation"
2. The "pri 2 escalation" policy points to a "pri2 day/night" rotation. That rotation sends to a "Pri 2 daytime" escalation during the day, and an "pri 2 nighttime" escalation at night.
3. The daytime/nighttime policies have rules. The first rule send to the on-call rotation. Then after a set time, it notifies a manager, then a senior manager, then the whole team. The amount of time between each step is dependent on the priority, and day/night. Which is why I need a separate escalation policy for each one

This is a huge limitation. If anything it will force me to build my integrations outside of teams, so that I can create teams based on priority, and use recipients in the create rules to route to the different teams. Which would force me to give more users full admin rights since they would need to be able to edit those integrations. Which is a bad idea, but I don't see another workaround with this limitation in place. If you are limiting escalation policies per team, I'll have to create more teams to get the behavior I want.

 Please reconsider this limitation, it is a severe limitation to the functionality of the product. The rules per escalation, and limits on repetition aren't as troubling but the limit on escalation policies is very problematic. 

[Shahroz Naeem]

I just finished migrating the organization to Opsgenie from PagerDuty and this pops up - I would appreciate an explanation on the decision to add limits - that too, of 7. If it was 20 or so, I wouldn't be worried, but 7 is too less. Also, would you be limiting the number of integrations per team as well next? Or the number of policies or actions we can have?

@Samir what you're explaining to Andrew is a work around to an existing functionality which was removed. That's a step down, which makes no sense. 

[Shaun Pinney]

Thank you @Samir Kafel OG for jumping in to answer questions here. 

And thank you to everyone who has added valuable comments and context on how you use escalations today. I wrote this post so I take full responsibility for not being clear and not including the why behind the change - I'm sorry it has disrupted your day and your Opsgenie use. 

I went back and added the Why and reasoning to the top of the post and I'll re-post it here as well. And as Samir mentioned, we also clarified the limits inside the post to answer some of the questions that have come up. Here they are again:

• Escalation policies per team: 7

• Escalation rule count per escalation policy: 10

• Escalation rule repeat count: 5

Why are we making this change?

We’re making this change to improve the performance and reliability of Opsgenie. In the past, if a customer with a large number of escalation policies, and multiple rules and repeats per escalation experienced a flood of alerts it could degrade the performance, and reliability of Opsgenie. In some cases, this has resulted in downtime and incidents, which are unacceptable for customers to experience. We feel these new limits enable every customer to continue to use escalations while also keeping the system up and running consistently.

Thank you for the feedback and we'll be here answering more questions as they come in. 

[Andrew Laden]

@Shaun Pinney Performance and reliability issues sound like bugs in the product. So rather then try to fix your own bugs, you are instead forcing your customers to limit how they can use the product.

If you have bugs, fix them. Don't make it your customers' problem. 

And you may want to talk to your customer base to find out how they use it as opposed to just a "feeling" that something will be acceptable. I "feel" that these new limits are not acceptable, and I am the one paying for the product.

There are so many better ways you could handle this. If you have customers who occasionally flood the system, maybe talk to those customers specifically and see if you can work with them to optimize their rules. You could find out more about how they use the system and help build something that would work for everyone.

This just feels like lazy product management, or a lack of resources (aka developers) 

[Todd Minnella]

Ok, I think I understand the new limits and, as other posters have suggested, this is an unwelcome change that may prompt an evaluation of alternatives for our alert handling.

We have an Opsgenie Team with several escalation policies (more than the limit) configured to handle specific alert sources differently (each has different responders associated, each with their own schedule).

Migrating from one team to several is doable (so as to stay under the escalation policy count limit), but it will require both moving an integration to the global level and creating many duplicate on-call schedules and rotations (since individual responders may have membership in multiple escalation policies). This will create a significant amount of work for my team and increases the complexity of our implementation.

Two questions:

1) Are these limits system-wide, or can they be adjusted on request for a specific customer/tenant?

2) If the answer to #1 is no (which is what I expect), is it possible to schedule a consultation with an Opsgenie solutions architect for help refactoring our current implementation? 

[Shaun Pinney]

Your points are very valid,  @Andrew Laden , thank you for the feedback. I am working with the Product team on this and will continue to post updates here as we have more information to share throughout the rest of the day today and into tomorrow 

[Cheyenne Gray]

This doesn't seem like the right solution. Would strongly recommend working with those specific large clients to reduce their rules causing performance issues rather than impose restrictions that make your product worse for everyone else. 

[Greg Markey]

When will you provide updated product pricing to reflect this regression in functionality?

[Daniel Winter]

Short lead time, no justification, no help for global administrators on big accounts (which teams are really impacted), very unprofessional - this change must be reconsidred!

[Smit Thakkar]

I am just waiting for a day when my company moves off of Opsgenie. Because of this change we have to rethink our alert routing strategy. 

[Smit Thakkar]

It clearly shows lack of customer centricity. I am pretty sure no company would ever pull such a stunt and survive. I wanna see how many companies abandon Opsgenie because of this. Would be good slap on the wrist on the Product Team there. 

[Daniel Winter]

A compromise could be to leave the number of escalations open or at a much higher limit and only limit the rule and repeat count. I suppose these rules can lead to a torrent of notifications if not applied carefully.

[Cody Yanniello]

@Shaun Pinney - similar to everyone above, this greatly affects our team as well.  On 9/8, you mentioned meeting with the product team to discuss this feedback, and that you would update this thread on 9/9 or 9/10. 

However, I don't see an update?  Can you please share?

PS - giving end users 14 calendar days to prepare for this change is asinine. 

[Shaun Pinney]

Hello all, thanks for your patience. I just updated the top of the post, and I'm posting the update below as well so everyone sees it. 

On Sep 7, 2022, we announced that new limits for escalation policies in Opsgenie would go into effect on Sep 21, 2022. We’ve since received feedback from many of you about the impact of the proposed changes. As a result of this:

• Our engineering team is re-evaluating how we approach escalations in Opsgenie overall and we will provide you with an update soon. 

• As part of this effort, we are reaching out to many of you to gather additional feedback about your escalation rules use cases.

• We will NOT add new limits to escalation policies on Sep 21, 2022.

We apologize for how the changes were previously communicated. Thank you for your feedback. We hear you.

If you have any additional questions, please drop them in the comments section below. Thank you for being an Atlassian customer.

[Andrew Laden]

Has anyone else gotten a meeting request from @Shaun Pinney to discuss this change? I've been trying to book time based on his request, but his calendar was booked for Sept, and October isn't even showing up yet. Did anyone else get a chance to talk with him?

[Shaun Pinney]

hi, @Andrew Laden I just emailed you, reply to that email and we can book time together, thank you!