Auto Close Dynatrace alert

Donny Gison February 26, 2022

We are using Dynatrace API to send Alerts to OpsGenie. Alerting works, but seems we are unable to auto close when DT alert is resolved. Any recommendations how to enable, auto closing based on DT resolution alert? 

 

Does it conflict with Schedule's acknowledgement? 

 

2 answers

1 accepted

0 votes
Answer accepted
Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 28, 2022

Hello @Donny Gison 

This is Mubeen. I will be happy to help you.

As I understand you want to automatically close the alert when it's resolved in Dynatrace.

You can achieve this by using the Dynatrace "Advanced" integration settings. Please note advanced settings for integrations are available only on the Standard and Enterprise plans.

From the Advanced Integration settings, you can set a "Close Alert" action which will close an existing Opsgenie alert based on its alias. The functionality to close an alert is reliant upon the “Alias” field of the alert. Here is the documentation to help understand the "Advanced Integration Settings


Regards,

Mohammed Mubeen
Support Engineer, Atlassian

Donny Gison March 12, 2022

@Mubeen Mohammed 

 

Seems to be set-up but not working? Thanks. 

 

Screen Shot 2022-03-13 at 1.13.21 AM.png

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 14, 2022

Hello @Donny Gison 
Hope your day is great,
As I can see from the screenshot shared you have just set the initial close alert rule. 

So as mentioned in my previous response, you can set a "Close Alert" action which will close an existing Opsgenie open alert based on its "alias" value.

 
[The alias is the unique identifier for "open" alerts. There can only be one "open" alert with the same alias in Opsgenie.]


So for the Close Alert rule to work, the below conditions are required to match, and the setup will be as follows.

1. You will initially configure a "Create alert" action. A "Create Alert" action will create a new Opsgenie alert if the condition set of its filter matches the data. You need to make sure to set a custom "Alias" value for this "Create alert" action. 

Create Alert.png
2. A "Close Alert" action will close an existing Opsgenie alert based on its "Alias" which means, for the "Close alert" action to work you need to have an Open alert with the same Alias field value configured in both "Create alert" and Close alert" action.
Close Alert.png

3. The actions are evaluated in order, each time the alert payload reaches Opsgenie it will be first evaluating the first action configured under your account. 


Ex: You have created "Create alert -1", "Create alert -2", "Close alert -1" and "Close alert -2" actions under your integration settings. 

Alert triggered from Dynatrace,
- It first checks the "Create alert -1" action if it matches alert will be created
- if not then It will go to the next action "Create alert -2" if this alert action is matching then based on its rules alert will be created. If there is already an alert open with the same information It will get de-duplicated.

- If not it will then route to "Close alert -1" at this point if this action matches then Opsgenie will check if there is any open alert with the "Alias" matching, and It will close that alert accordingly and It will not check any further with the last action "Close alert -2"

For any action set up for integration, an Alias value must be defined and matched exactly in order for the action to be executed.

Hope this information is helpful, please let us know if you have any further questions. I will be happy to assist.

Regards,

Mohammed Mubeen
Support Engineer, Atlassian

 

Like Donny Gison likes this
Donny Gison March 15, 2022

Hi @Mubeen Mohammed , 

 

That's helpful. Made me understand Actions more. May I please seek further feedback.

In your example the Alias to match is hard coded. What if it changes like DT problem number. 

 

DT Alert Creation Message - 

[PROD] SmartQ UW: OPEN Problem P-220343162

 

DT Alert Closure Message - 

[PROD] SmartQ UW: RESOLVED Problem P-220343162

 

May Alias values is likely not right :) 

Screenshot 2022-03-15 174147.png

Donny Gison March 15, 2022

Apologies :) I think I misunderstood the value for Alias.  I have updated and trying the below . 

 

Just waiting for the next alert to test it out.  Opted to not use Create Alert, for now, to not mess with the count of alerts. 

Screenshot 2022-03-15 174931.png

Donny Gison March 15, 2022

Unfortunately this did not work as well. 

Won't the DT Alert for Resolved be logged somewhere as well? We seem to not be getting one. The closure stopped for us sometime Feb. Now I am wondering if its the Integration API. Not sure 

 

Here is a sample log when the Auto Closure from a Resolved DT alert was still working when received, hope this helps. 

 

Screenshot 2022-03-15 191949.png

Donny Gison March 15, 2022

The creation seems to work. We just had 2 DT alerts today that Opsgenie was able to tag duplicate. 

Screenshot 2022-03-15 192654.png

We got 2 DT alert emails with different email subjects but the same body of text,  Opsgenie was able to have them in one alert only, just increased the count to 2. 

Creation works including duplicates but closure doesn't.  I still don't fully get Alias haha 

Here's the Create set-up. Maybe I need to define exactly the same fields for Close alert? 

Close just have Alias, User and Notes. 

Help will be appreciated. 

 

Screenshot 2022-03-15 193009.png 

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 16, 2022

Hello @Donny Gison 

I can help explain the concept of "Alias", as mentioned earlier The Alias field is a user-defined unique identifier for "open" alerts. 

From the last deduplication message screenshot you sent, you can see it has the Message: "[Dynatrace]: [PROD]: SmartQ: U/W ................... low disk space"

And you can see the count increased to 2. You got this as "Alias" because you have set the "Create alert" action alias field as "[Dynatrace]: {{message}}".

Since the alert was received with the same alias value and it matched with the "Create alert" action so it got deduplicated. 

Please note: As mentioned earlier in my previous response,
The payload received from Dynatrace will be checked in the order your alert actions are created.
Screenshot 2022-03-16 at 12.03.46 PM.png

As per the order, the first section is "Create alert" when the alert payload comes and it matched with the first action "Create alert" it will create the alert. So you need to send a different payload that matches only with "close alert" to be able to close the alert which is already created with the same "Alias" value. 


Incoming payload from Dynatrace > -- matches with -- > "Create alert" = Alert will be created.

Incoming payload from Dynatrace >-- matches with -- > "Create alert" and an alert is already open = It will be deduplicated

Incoming payload from Dynatrace >-- matches with -- > "Close alert" and an alert with the same alias value is already in open state = Alert will be closed. 

For your testing try to set the "alias" values in both create alert and close alert action as I specified in the previous response screenshots. 

Hope this answers your queries. 

Regards,

Mohammed Mubeen
Support Engineer, Atlassian



Donny Gison March 17, 2022

Hi @Mubeen Mohammed

Thanks for continuing to respond. I am slow in understanding Aliases I think. Let me try the below. 

 

  • Create Alert 

Create Alert.png

 

  • In the sample deduplication, the Message are not actually identical.

1st DT Alert has “Asia ESB”

DT alert for Create 1.png

2nd DT alert that became deduplicated has “SmartQ UW”

DT alert for Create 2.png

Opsgenie was successful to create and deduplicate even with some words different on the description of the 2 DT alerts. 

 

Now, for Close alert, DT sent these 

For the first alert - different is just the word "RESOLVED" instead of "OPEN"

DT alert for Resolved 1.png 

For the second alert - different is just the word "RESOLVED" instead of "OPEN"

DT alert for Resolved 2.png

 

This confused me on how Alias works. Deduplicate was able to work with different words on the Description of the 2 DT alerts received. But Close was not able to with just "Resolved" as the difference.  I might be understanding Alias incorrectly. 

 

We tried adding a filter of looking for "Resolved" in the Close alert because the default of just Alias someshow stopped working in February.  

Close Alert .png

 

 

Maybe given the DT alerts we receive you can be very specific on what may work as Close alert set-up? To make it work again. And other Action set-up I need to do based on what we get above. 

Also, is there a way to confirm that the "Resolved" alert from DT did reach Opsgenie to go through the Actions? This is one thing I can't find so far. 

 

Hope you continue to help me solve this. Thank you. 

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 17, 2022

Hello @Donny Gison 
No worries I can help clarify your queries with Alias. 


For the alerts which got deduplicated with the Alias field and from the screenshot I see you have just set "{{alias}}". To help understand how you got two different alerts message content was de-duplicated, I have provided the information in my previous responses which is a common setup for most of the Opsgenie configuration.

But your recent query requires checking configuration and logs within from your Opsgenie instance because it will be a unique setup/configuration for individual instances. You can reach out to me at https://support.atlassian.com/contact/#/ so I can further check account-specific details to help further. 

Also, I would like to share a specific example configuration that will allow you to create and close an alert, so you can test this further to understand/implement it accordingly.

For our example, let's just consider you are getting the following data for an instance state change and you need to set Create and Close actions accordingly.

Ex:
Server.abc is currently not responding. (Create alert)
The issue with Server.abc is resolved now. (Close alert)

Initial setup 
Screenshot 2022-03-17 at 3.52.14 PM.png

As per this setup when I receive the payload as "Server.abc is currently not responding" it will create an alert for me as I have set a message containing "not responding" as a condition to get the alert to create.


Now the "close alert" configuration setup in relation to the alert created by the above "Create alert" condition. 

Screenshot 2022-03-17 at 3.53.15 PM.png

As you can see from the screenshot I have set the condition as a message that contains "resolved now" should match and with the same alias value if there is any open alert then close it. 

Hope this clarified the Alias role and alert conditions. You can try to test with the exact value to understand how it works. 

Just want to add another payload example in relation to the examples shared above to clear your understanding.

Let's say you have sent the resolved payload a little differently as follows "The server.abc which was not responding that is resolved now"

This technically resolves the alert as it has "resolved now" in it, but it will not instead it will de-duplicate the alert. Because before the alert condition match the "Close alert" action it matched with the "Create alert" action "not responding" hence it de-duplicated


I have specified in my previous responses Payload checks the alert action in order, the first action it matches will be executed and it will stop from there. 



Hope this example clarified your query with "Alias". 

Regards,

Mohammed Mubeen
Support Engineer, Atlassian



Donny Gison March 17, 2022

Hi @Mubeen Mohammed , I raised 

https://getsupport.atlassian.com/servicedesk/customer/portal/45/OGSP-86337

to investigate the logs and confirm that DT Resolved alerts do reach our Opsgenie instance. 

Tried defining an Alias of [PROD] for both Create and Close, unfortunately it did not work. 

Prod Create Alert.png

 

Prod Close Alert.png

 

The DT alerts 

 

For Create

PROD Open.png

 

For Close, but seem to not consumed by Opsgenie 

 

PROD Resolved.png

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 17, 2022

Hello @Donny Gison 

Yes, I got the ticket you have created, I will further help with the ticket you have opened :) 

Also to clarify on the screenshot you shared above for the "create alert" action rule you have mentioned the filter condition as "Match all alerts

Prod Create Alert.png


As mentioned in my previous responses all the Alert actions for CREATE or CLOSE will be checked in order, if the alert payload matches with the first one then the next alert action will not be checked. 

Since you have set the alert filter as "Match all alerts" it will match with all alerts incoming payload, including the "Resolved" payload, hence no "Close Alert" actions were executed.

Instead of "Match all alerts" you can set a specific value as one of the alert conditions to test it again. 

Regards,

Mohammed Mubeen
Support Engineer, Atlassian

Donny Gison March 18, 2022

Noted @Mubeen Mohammed

We can explore the Match alert options for Create next. For now, let's analyze first the Resolved alerts from DT via the raised ticket. If Opsgenie is receiving them, and if possible to have a view of what we are receiving on the API. Thanks.  

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 18, 2022

Sure :)
I have already sent the required information on the ticket to further investigate this.

Regards,

Mohammed Mubeen
Support Engineer, Atlassian

Donny Gison March 26, 2022

Thanks for all the help @Mubeen Mohammed , our DT team has switched to webhook API and closure is now working. We'll just have to work on our Policies to ensure it's being routed to the right schedule and people. 

Jason Selby April 19, 2022

Was this an issue with Dynatrace or Atlassian ? I am also experiencing it

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 19, 2022

Hello @Jason Selby 
This is Mubeen. I will be happy to help you.

If you are currently facing an issue with closing the Opsgenie alert when the original alert is resolved in Dynatrace, it is possible that your Opsgenie configuration settings for the Dynatrace integration are not configured correctly or it is also possible that the alert closure payload from Dynatrace is not reaching Opsgenie to work as per your Integration settings. 

- You can cross verify the validation options discussed in this ticket to check accordingly.

- From the Opsgenie debug logs section validate if the Dynatrace alert closure payload is reaching Opsgenie. (If not you are required to work with the Dynatrace admin who is currently working on your Dynatrace configuration)

- If you are able to receive the closure alert payload from Dynatrace to Opsgenie(By checking in Opsgenie Debug logs) then you can validate the "Close Alert" action in the Dynatrace integration settings of your Opsgenie account. 

 

Hope this answers your queries. 

Regards,

Mohammed Mubeen
Support Engineer, Atlassian

Jason Selby April 21, 2022

Mohammed,

The issue i am having has been introduced recently, i have checked everything above and it is setup in the correct way, the close just doesnt work. Would it be possible to have a support call?

Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 21, 2022

Hello @Jason Selby 
This requires checking configuration and logs within your Opsgenie account because it will be a unique setup/configuration for individual instances. You can reach out to us at https://support.atlassian.com/contact/#/ with the complete integration/alert details so we can help check/investigate the issue accordingly. 

 

Regards,

Mohammed Mubeen
Support Engineer, Atlassian

0 votes
Mubeen Mohammed
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 27, 2022

Hello @Donny Gison 
Thank you for the confirmation and I am glad it is working as expected for you now.

Regards,

Mohammed Mubeen
Support Engineer, Atlassian

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events