Azure Resource Health integration does not close alerts

Question

I've configured Azure Resource Health integration, and noticed that alerts are not closed automatically even though they have Resolved activityLog status.

I looked at the Advanced configuration of the integration, and found out that the Close action checks the main status field of the message to be "Resolved". Given the Microsoft documentation about the resource health message payload, the Close action is never going to be triggered, since the main status is always "Activated" for this kind of message. Only the Activity Log Status is going to change, and here is an example of such a message:

{
  "schemaId": "Microsoft.Insights/activityLogs",
  "data": {
    "status": "Activated",
    "context": {
      "activityLog": {
        "channels": "Admin, Operation",
        "correlationId": "...",
        "eventSource": "ResourceHealth",
        "eventTimestamp": "2021-08-18T15:23:00.2603901+00:00",
        "eventDataId": "...",
        "level": "Informational",
        "operationName": "Microsoft.Resourcehealth/healthevent/Resolved/action",
        "operationId": "...",
        "properties": {
          "title": "Unknown",
          "details": "Unknown",
          "currentHealthStatus": "Available",
          "previousHealthStatus": "Unknown",
          "type": "Unknown",
          "cause": "Unknown"
        },
        "resourceId": "...",
        "resourceGroupName": "...",
        "resourceProviderName": "Microsoft.Resourcehealth/healthevent/action",
        "status": "Resolved",
        "subscriptionId": "...",
        "submissionTimestamp": "2021-08-18T15:23:14.0751627+00:00",
        "resourceType": "MICROSOFT.SQL/SERVERS/DATABASES"
      }
    },
    "properties": {}
  }
}

I wanted to change the behavior of Close action, but there is no way of selecting Activity Log Status for the filter conditions...

Also there is no Activity Log Status field in draggable items:

even though it's present in extra properties, but with a spelling mistake:

I know that I can still use {{_payload.data.context.activityLog.status}} but still I consider this as a bug.

Skyler Ataide · Answer

Hi Michal,

Apologies for the delayed response. According to this integration's settings, the payloads are checked against Status filter. For Activated status; by default, Create alert action gets applied and for Resolved statuses; Close alert action gets applied. Since all of your activity log alerts are coming into Opsgenie with the status always set to Activated in the payload, this is the reason that duplicate alerts are being created.

From what I understand, it does seem incorrect that the default Create Alert and Close Alert actions would behave in this way considering that the Status included in the payload being sent from Azure is always equal to "Activated" in the use case you've described.

Are you able to raise a support ticket with Opsgenie support using this link here: https://support.atlassian.com/contact/#/ so that we can dig into this issue deeper?

We may need the ability to access your account instance to see if we are able to reconfigure the filters in these actions to Create and Close alerts that come in from your Azure Resource Health integration correctly. There is also a potential workaround here that can be deployed, however, it will be best to see if we can get the default integration actions to work appropriately first. If the default integration can not work due to the reasons you've described, then this problem could indeed be classified as a bug, in which case we will want to escalate the full details of the issue to engineering so that it can be fixed.

Thank you,

Skyler

Michal Keder_ix6s1s · Answer

Hi Skyler,

My Azure Resource Health Close Alert action is set to default as follows

Also Azure Resource Health rule is configured as it should be

Alerts are coming to OpsGenie, with event statuses Active, Updated and Resolved. I can see them in OpsGenie debug logs, but as I already stated, the default Filter for Close action is not going to work, since it checks the Status field of the message, and Microsoft documentation states that it's always set to Activated no matter the internal activity log status:

Currently the result is that Count of the alert with the same resourceId is increased but the alert itself is not closed even though the activityLogStatus is Resolved.

Skyler Ataide · Answer

Hi Michal,

Thanks for reaching out, and thank you for providing this information! Are you able to reconfigure your Azure Resource Health Close Alert action to filter for alerts where the Resource Id has a Status of "Resolved"? Also, is there an Azure Resource Health rule for when the Status is Resolved on the Azure side, and does it have the Opsgenie Web-Hook? Proceeding with these steps, if your Azure side sends out the Resolved status messages to Opsgenie, then it should be able to automatically close these alerts. Regarding the extra properties Status: {{activitLogStatus}} spelling error, thank you for pointing this out; I will reach out to our engineering team to see to it that this is corrected. Hope this helps, and please let me know if you have any further questions.

Thanks,

Skyler