Unable to resolve SumoLogic incidents from Opsgenie

[aryanp]

Hi team,

I have created an integration from SumoLogic to OpsGenie wherein when an incident is created in SumoLogic a corresponding alert  is being triggered in OpsGenie.

However, when an alert is closed from OpsGenie, the corresponding incident created in SumoLogic is not getting automatically resolved and we have to manually resolve the SumoLogic incident.

Is this an expected behaviour?

I was looking for something like the incident in SumoLogic should be automatically resolved when I close an alert form OpsGenie.

Is this possible?

Do let me know if any other details is required from my end.

[Nick H]

Hi @aryanp ,

The Sumo Logic integration is a bit limited, and can only "send data" back to SM when an Opsgenie alert is closed:

I'm not a SM expert, nor really know what data is actually being sent from Opsgenie - but if there is some sort of automation that can be configured on the SM side when this data is received, I'm sure it could close the SM incident.

But again, not sure SM has this type of functionality like Jira or JSM currently have with automations.

[aryanp]

Hi @Nick H 

Thanks for the confirmation response.

There's one more issue that I am currently facing with the SM and Opsgenie integration.

I have basically set an alert to be generated in OpsGenie whenever an incident changes state in SM.

For some time this integration works and the corresponding team users are notified of the alerts. But after sometime , the notification trigger seems to stop working.
Basically, an incident creation in SM is not triggering a notification to the users.

I have also configured an incident for the particular team.

I can see this in the Opsgenie alert activity log,



The notifications were working fine for sometime and then it stopped working.
No config changes were made.

Also, earlier when everything was working as expected , for each SM incident, i could see two alerts being created in Opsgenie.

But now I am able to see only one alert.
Only the associated alert (A) . 

Can you assist here?

[Nick H]

Hi @aryanp ,

It appears that the team receiving these alerts also has an incident rule configured to automate the SM alert into an incident.

When an alert is automated into an incident through an incident rule, the initial alert becomes an associated (A) alert - which in turn triggers a responder (R) alert (or multiple alerts if the service tied to the incident rule has multiple responders tied to it).

Any alert that is created after the incident is raised and matches the incident rules will also be associated. Important to note that only the responder alert(s) will notify users, while the associated alerts simply add to the incident without notifying:https://support.atlassian.com/opsgenie/docs/associate-an-alert-with-an-incident/#Associated--amp--Responder-Alert-Behavior

Here's an example of an incident rule I have for my Community Team:

Incident rule is tied to this Service, which adds other responders to the incident:

Alert created >> matched incident rule >> triggered associated (A) and responder (R) alerts:

[aryanp]

Hi @Nick H 
I'm still seeing only the Associated alert in the Alerts dashboard.

Sharing my sample incident rule,


Here my incident rule is tied to SumoLogic Alerts service.




Added Hyperexec as the Owner team in Responders.

However, I am still not receiving the notifications for Hyperexec team users.

Only the Associated Alert is being generated.



Am I missing something here?

As mentioned earlier , the responder alerts were getting generated and the notifications were also triggered. But the notifications stopped working all of a sudden and no Responder alerts are generated.

[Nick H]

Seems like the alert(s) is still being associated with an existing incident - whether the incident remains in an open state, or a resolved state. You should be able to check which incident that alert associated with here:

It's not recommended to have an incident rule set to Match All like you shared - since it acts like a catch-all. Alerts cannot create an incident per alert when that is configured within the filter, so they'll continue associating to an existing incident.

There is an open feature request for that type of functionality that can followed, watched, and voted for here: https://jira.atlassian.com/browse/OPSGENIE-73

If you were to adjust that incident rule filter to include some conditions - or even delete it - I believe you'd notice the alerts acting normal and notifying your users as expected.

[aryanp]

Hi @Nick H 

Thanks for the information.
After removing the incident rule, the notifications seem to be working.

However,
There's still another issue that I'm observing.
When an open incident in SumoLogic is getting resolved[manually/automatically] , 
the triggered open alert in Opsgenie is not getting closed automatically.

Attaching the close alert integration rule that I have created,



As a result of this, new notifications are not getting triggered from Opsgenie.
And the SumoLogic incident is getting mapped to the already opened alert in opsgenie.

[Nick H]

Hi @aryanp ,

Glad notifications are working as expected now.

As for the SumoLogic alert not closing the Opsgenie alert, it could be due to your Create alert action filter. Maybe it's acting like a catch-all with the filter set to Match All conditions?

Regardless, try reviewing the Logs tab to see what data / request is being received by Opsgenie when the SumoLogic alert is being closed.

If no data / request is being received, then my guess is SumoLogic is not sending these to Opsgenie. If it is being received, review what might be happening - whether the alert is being deduplicated, the alert is not matching on the Close alert action's filter / condition, etc.

This article may help as well. Although it's for when alerts are not being created, the troubleshooting steps would be the same to review.

If Opsgenie is receiving the close request, the two logs you most likely want to review are the Processed incomingData and Received integration request logs: