Incident creation based on alert count

[Urban Jurca]

Hi team,

we are trying to come up with an automated incident creation rule that fires based on alert count (grouped by tags?).

In a perfect world this is something we would like to achieve:

- we have an internal DB service

- all alerts related to the DB service are tagged by service: DB service

- we would like to automatically open an incident if we have e.g. 3 open alerts where tag == DB service; or an alert with tag == DB service has a P1 priority.

Automatic incident creation based on priority works fine, but any more elaborate rules are not natively available. How do you solve this challanges?

Best,

U

[yaniv]

I would love to have a count added as a condition to this view, it would save us having to write a REST API call as the webhook feature is only triggered once.

[Volkan]

Hi Urban,

I assume you do not have any issues with setting up your service rules based on the alert priority or tags for automated incident creation but the issue you'd like to overcome is to have Opsgenie trigger a new incident based on the count of the associated alert.

The automated incident creation based on the alert count is unfortunately not one of the default options you can configure within your services. However, any custom solution that could be implemented through Opsgenie REST API or Webhook may be a workaround for you here in this case.

If you possibly choose to develop a custom script that counts the alerts having a specific tag, it will be possible for this script to create a new incident through Incident API. It is also doable in your custom solution to get the alert deduplication count via Opsgenie API or Webhook/OEC integrations. An outgoing webhook request can be sent to a URL you will provide in Webhook integration settings when there is a new alert with a specific tag like "DB Service". Please note that the outgoing integrations like Webhook integration are only available in Standard and Enterprise plans.

On the other hand, we also have raised a feature request to our product team to make this feature available natively as there are some other Opsgenie users requested the same before.The whole incident management concept is getting more advanced with the new feature sets and this might be one of the enhancements that may be delivered in the future. I will update our internal feature request with this community post to get our product team's attention on this one and I hope they will be making an analysis for this in one of our future development sprints!

Hope this helps!

[Mokhlich Ali]

Hi Volkan, you mentioned that "An outgoing webhook request can be sent...when there is a new alert".

I have done exactly so that a newly created alert can create an incident in Opsgenie.

However when another alert with the same alias comes in, as expected only the count increases and no new alert is fired, but unfortunately therefore my trigger on my webhook does not fire.

Can you suggest any other ideas? Will it be possible for Opsgenie to send a webhook request for when the count increases?

Many Thanks

[Jonas Martinez]

Hello @Mokhlich Ali

It sounds like you got it right, when Opsgenie receives an alert with the same alias as a previously-open alert, it deduplicates into that original alert and increases the count. 

https://support.atlassian.com/opsgenie/docs/what-is-alert-de-duplication/

Because of this, if you are triggering a webhook out of Opsgenie, it's only triggered when an alert is newly created, and when we de-duplicate, it doesn't count as a newly created alert.

My suggestions would be to....

- Make sure that your original alert is closed ASAP so that when we receive that second payload with the same alias, it just creates a new alert and that should trigger your webhook.

- Other than that, you could set up your "Create alert" action to use a different alias for each alert that comes in, so every alert that Opsgenie receives will be treated as a new alert and not de-duplicate. You can do this by deleting the "alias" field of your "Create Alert" action in the advanced integration settings.

If you have any further questions, feel free to create a support ticket for the Opsgenie support team via this link: https://support.atlassian.com/contact/

[Fernandez, David]

Hi @Volkan

any news regarding a native implementation of this feature ?

[Darryl Lee]

Hello @Fernandez, David ,

This is Darryl, I am filling in on behalf of Volkan.

As I just checked, this feature request hasn't been implemented yet and we have created a public FR ticket over OPSGENIE-595.

Please vote it and set yourself as a Watcher for this request to receive the latest update.

---

Thank you very much for your feedback in helping us improve Opsgenie and we are sorry for this long waiting on this feature request.

Kind regards,
Darryl Lee
Support Engineer, Atlassian