nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

SCOM alerts going from Warning to Critical doesn't get into OpsGenie

Michael Stefansen

Hi Community

I'm new to OpsGenie and has been handed the responsibility to integrate it with our System Center Operations Manager (SCOM) monitoring tool.

I've setup all alerts to be send to OpsGenie, but I only notify my on-call rotation with SCOM alerts with Severity "Critical".

This means that alerts of severity "Warning" should be send to OpsGenie, but should not be send out to the on-call rotation, like forexample a disk hitting a SCOM threshold that we have decided as being a warning threshold. But when that threshold reach a severity "Critical" because someone didn't react to the warning alert, I need OpsGenie to send that alert to our on-call rotation.

I would expect that SCOM closes the warning alert in SCOM, as the severity has changed to Critical and then sends a new alert, with a new ID, to OpsGenie. But that doesn't happen today - and I struggle to figure our why. Anyone here who has experienced the same?

Thanks in advance.

Accepted answers

martin.simonsen

Hi, Im just adding to this.

SCOM is changing the state of the alert from warning to critical not the initial eventID. How should this be handled in OpsGenie?

All comments

Skyler Ataide

Hi there Michael!

I'll be happy to see if I am able to provide assistance with this issue! The SCOM>>Opsgenie integration works in a way such that when an alert is created on SCOM, an alert is created in Opsgenie automatically through the integration. If SCOM does in fact send a new alert with a new ID/Alias to Opsgenie when the severity gets changed from Warning to Critical, then you would be able to use routing rules to send these new alerts with Severity Critical to the appropriate On-Call Team.

However, if I understand correctly, this is where you are having trouble since new alerts are not being created when the severity changes in SCOM? An initial suggestion that I have here would be to see if you are able to set up separate Create Alert actions that filter for different severity level alerts that come into Opsgenie from your SCOM integration. For example, the first Create Alert action in Opsgenie can filter to match all conditions where Severity equals Warning, and then you can set those alerts to have a lower priority of P3/P4. Then, the second Create Alert action can filter to match all conditions where Severity equals Critical, and set those alerts to a higher priority (P1/P2), which in turn can be routed to your On-Call Team based on the higher priority. This all can be configured from the Advanced Settings page for your SCOM integration.

Hope this helps!

Best,

Skyler

martin.simonsen

Hi, Im just adding to this.

SCOM is changing the state of the alert from warning to critical not the initial eventID. How should this be handled in OpsGenie?

Michael Stefansen

Hi Skyler

Thanks a lot for your willingness to try to help out here, it's highly appreciated :-)

I've made a test environment and connected that to our SCOM test environment, where I would like to test the use of OpsGenie Prioritization. I suppose I'll need to just send everything from SCOM (Informational, Warning & Critical) alerts, then create 3 "Create Alert" covering the 3 severities being send over from SCOM, like in below screenshot:

Then for each of these I have the opportunity to set the requires OpsGenie priority (P1, P2, P3, P4 or P5)

This one if for the Critical alerts, then I would assume that all SCOM alerts of severity critical will get the Priority "P3 - Moderate" when they are moved to OpsGenie.

To test this I just set my routing to send out everything, to verify that critical alerts has the P3 priority, but I never get any critical alerts :-(

I do get some Warning alerts though, which is prioritized as P4, which is set under "Warning alerts" and is working just fine, but it's doesn't over all warnings from SCOM :-(

Can you guide me a bit more, there is definitely something which isn't configured correctly :-)

Thanks in advance for your time.

Skyler Ataide

Hi Michael,

Happy to help, and thank you for providing the additional information/screenshots! The three Create Alert actions that you have set up to match different priority levels in Opsgenie look good based on the provided screenshots. In order to investigate why no critical alerts and only some warning alerts are coming into Opsgenie, you can check the logs under the Settings>>Debug>>Logs page in Opsgenie. You can check the Logs to ensure that Opsgenie is receiving the integration requests from SCOM, and to make sure matching actions are found to create the corresponding alerts in Opsgenie.

From the Logs page in Opsgenie, you will want to identify the Received Integration Request from SCOM, and then expand the incoming data to get a full view of the payload that is being sent from SCOM to Opsgenie. Here is the sample payload that Opsgenie recognizes officially from SCOM:

{
"owner": "np",
"lastModified": "12/24/2015 11:47:16 AM",
"resolutionState": "New",
"timeRaised": "12/24/2015 11:47:16 AM",
"resolutionStateLastModified": "np",
"workflowId": "{7eba60fd-b179-69a7-3897-47b6753601f2}",
"category": "Custom",
"alertId": "{2ba87d56-a7af-4b42-bdcc-eb18486bd8cd}",
"alertName": "Alert for event 999",
"priority": "1",
"severity": "2",
"createdByMonitor": "false",
"repeatCount": "0",
"alertDescription": "np",
"managedEntitySource": "WIN-RQTU8UB5TU5.opsgeniescom.com"
}

Based on the sample payload that Opsgenie expects to receive through your SCOM integration, my prior response suggestion to match all conditions where Severity equals Critical/Warning/Informational may need to be changed to equal 2, 1, or 0, since these are the values that Opsgenie expects to receive in the payload. These numbers correspond to the different SCOM Severities; where 2 = Critical, 1 = Warning, and 0 = Informational. Additionally, based on the conditions of your Create Alert action, in the integration payload there will need to be "resolutionState": "New" so the integration can be processed.

If you are still unable to identify why these alerts are not being created in Opsgenie after troubleshooting the logs for your SCOM integration, may I ask if you are able to raise a support ticket with Opsgenie support using this link here: https://support.atlassian.com/contact/#/

If this request is raised as a support ticket, you may be asked to grant temporary access to your Opsgenie instance to our support team, so that we can take a closer look into this issue to help make sure that you get this integration all set up correctly.

Best,

Skyler

Michael Stefansen

Hi Skyler

Thanks for pointing me in the right direction again :-)

The issue was severities and what actual severity OpsGenie expects. I thought SCOM was sending Critical as the severity, but it doesn't, it sends "Error" for critical alerts. Changing my "Critical alerts" to Severity equals "Error" made it all work like a charm :-)

again a huge thank you for your guiding and suggestions, my issue has been solved - YAY :-)

Have a great day.

Skyler Ataide

Hi Martin,

In sounds like you will need to adjust the filter(s) in your SCOM Create Alert actions to match the desired data received in Opsgenie from your SCOM Integration. If the payload received from SCOM matches a Create Alert action, and Opsgenie notices that there is already an open alert with the same alias, then that alert will be de-duplicated in order to prevent alert notification fatigue. Here you can find our documentation that covers Advanced integration settings: https://docs.opsgenie.com/docs/advanced-integration-settings and here our documentation covers more details on configuring your action filters: https://docs.opsgenie.com/docs/filters

Best regards,

Skyler

Michael Stefansen

hi Skyler,

We solved the issue by removing the "AlertID" alias on the alerts where we could have multiple alert states. Like forexample Logical Disk Space is low.

By doing that we'll get a new alert in Opsgenie everytime the Logical Disk alart is changing state and that is what we want.

The downside of doing this is then, that the alerts wont close automatically, as OpsGenie doesn't have a unique identifier to close the alert from, when it's changing state. So we add another manual step to the alert handling, but we live with that for now :-)

Thanks again for all your input, it guided us in the right direction :-)

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

opsgenie