Handle SP API threshold limit

[pedro.gonzalez.castro]

Hello all,

We are using Opsgenie to create & update some Incidents in the Status Page. For example, if a certain Opsgenie problem is created, it will create an incident in the Status Page and if the problem is closed in Opsgenie, the incident will be resolved in the Status Page.

Due to the nature of our application, sometimes these problems are a lot and they are quickly generated (or closed). Once, we saw that all problems were closed in Opsgenie but some of them were not resolved in the Status Page, and we had to close them manually. 

After some digging, we saw these logs in the Opsgenie problems that were not resolved in the SP:

Could you please let me know if there is a way of handling this in the Opsgenie/SP integration? Something like doing another attempt some seconds later.

Thanking you in advance,

Pedro Gonzalez

[David Bakkers · Accepted solution]

Hello @pedro.gonzalez.castro 

If you Google 'Statuspage API limits' you will see prior threads such as this one in this forum dealing with the same topic. As per the documentation, requests must keep within the 1 request / second limit, as measured on a 60 second rolling window (IE. 60 requests per minute) for the Manage API.

It seems like your OpsGenie incident change process is happening too quickly and is creating too many requests for the native OpsGenie / StatusPage integration to keep within that limit, but I would have assumed that it had a built-in 400 series error detection and retry function to ensure rejected requests are always retried and go through in the next 60 second window. If you are seeing that the rejected request to SP is NOT being retried and completed at all, you may have to raise a bug with Atlassian.

In the interim, think about adding some delay to your OpsGenie incident management "application" or process to slow the pace down a little; let's be honest, opening and closing a new incident every single second, seems like overkill somewhere.

[pedro.gonzalez.castro]

Thanks, I will definitely raise a bug with Atlassians, seems to be the right move.

EDIT: Unfortunately, we can not add delay nor control the frequency of the alert/incidents updates.