Heartbeat monitoring of CloudWatch integration

[Piotr Bzdyl]

Hi,

I've configured a CloudWatch integration so that CloudWatch alarms are delivered to OpsGenie. I have a CloudWatch alarm sending alarm events to SNS topic and I've configured OpsGenie CloudWatch integration endpoint as an HTTPS subscriber.

Now I would like to configure a constant monitoring of this integration to ensure that it works especially at the moment where a severe incident happens and CloudWatch alarm needs to be delivered to OpsGenie so somebody is paged.

My idea was to configure a "cron" task to publish a "ping" message event periodically on the same SNS topic so it's delivered to OpsGenie as a heartbeat. I want OpsGenie to treat these events as heartbeats and page me when a heartbeat message is not received for some longer period of time (as it might indicate that alarm delivery from AWS to OpsGenie doesn't work properly). Is it possible?

I know there is a feature called HeartBeat Monitoring. I thought that maybe I could create an heartbeat endpoint and configure it as another HTTPS subscriber on the same SNS topic. However, heartbeat endpoint uses a different "protocol" - it requires HTTP authorisation header (whereas CloudWatch integration includes the API key in the URL).

Am I missing something? How can I make sure that the alarm delivery works and if it stops working I get paged by OpsGenie?

Best regards,

Piotr

[Nick H]

Hi @Piotr Bzdyl ,

That's a very cool use case that I don't believe I've seen before, but might be possible!

Like you mentioned, Heartbeats are typically configured with a cron to ping Opsgenie depending on a certain time interval, and if Opsgenie does not receive the ping >> create an alert.

Although our documentation does require HTTPS subscription for the integration: https://docs.opsgenie.com/docs/amazon-sns-incoming-integration#setup-subscription-on-amazon-sns

SNS subscriptions do have the option to Email as well: 

So I suppose you could configure the same email of the Heartbeat here and it might meet the use case needed:

^ With both this and your normal SNS integration, Opsgenie would create an alert when this is sent successfully, as well create an alert when it is NOT - since the heartbeat would expire.

Note entirely sure if this is what you need / helpful, but let us know if you have any questions, issues, etc.

[Piotr Bzdyl]

Thank you @Nick H for your suggestion. I will take a look. With this solution I can have a single SNS topic for real alarm messages from CloudWatch alarms and also send to the same topic my heartbeat messages.

I have a concern though - these ping messages would be delivered also to the OpsGenie CloudWatch integration HTTPS endpoint - wouldn't them interfere somehow with the alarm delivery? Can I send just any payload to the CloudWatch integration HTTPS endpoint and they will be acknowledged to SNS?

Also I don't want the heartbeat messages sent through the HTTPS subscription to the OpsGenie CloudWatch integration to generate any alerts. Would such a setup be supported by OpsGenie/Atlassian?

[Nick H]

Hi @Piotr Bzdyl ,

Sorry for the delay here! I lost track of this so I do apologize.

You mentioned, "I have a concern though - these ping messages would be delivered also to the OpsGenie CloudWatch integration HTTPS endpoint - wouldn't them interfere somehow with the alarm delivery?"

I don't believe it would be since you'd be sending a request to both the integration and heartbeat through 2 different methods.

For this use case, I believe you'd want to have a Topic/Subscription for each method. The SNS integration would be "pinged" through HTTPS, while the heartbeat would be "pinged" through Email:

You also asked, "Can I send just any payload to the CloudWatch integration HTTPS endpoint, and they will be acknowledged to SNS?"

I'm not entirely sure what you mean by this. Opsgenie is expecting it in JSON though. The integration needs to use HTTPS as mentioned in Step 2 here: https://docs.opsgenie.com/docs/amazon-sns-incoming-integration#setup-subscription-on-amazon-sns

When creating the integration's Subscription and selecting HTTPS, you will want the Enable raw message delivery checkbox UNSELECTED to ensure this is sent as JSON:

Hope this helps! But let us know if you have any other questions, issues, etc.

[Piotr Bzdyl]

Hi @Nick H ,

I finally got to test your suggestions to use OpsGenie Heartbeat email for an email subscription on the same SNS topic as the CloudWatch OpsGenie integration

Unfortunately, this won't work as Heartbeat email address doesn't give me access to the contents of the emails retrieved and I cannot even confirm SNS email subscription.

What's more, this still would be a suboptimal solution as I wouldn't be testing/monitoring 100% of the CloudWatch integration path as I would be monitoring CloudWatch alarm -> SNS topic -> email subscription -> OpsGenie Heartbeat path, and my goal is to monitor CloudWatch alarm -> SNS topic -> HTTPS subscription -> OpsGenie CloudWatch integration.

Thus I think this is a lacking feature in OpsGenie CloudWatch integration - it would be ideal to have this integration to support also "Heartbeat" feature directly.

Best regards,

Piotr

[Samer Shami]

Hi. Sorry to bring up an old thread. I am interested in finding a solution to this.

I need to alert when something doesn't appear in a SNS topic for a certain period of time. For example, if I am using AWS Backup and the SNS topic doesn't receive either a success or failure message for a certain amount of time, then OpsGenie should generate an alert.

So Heartbeat is perfect for this. However, I can't use the API version of Heartbeat because I can't send a header in the SNS HTTPS endpoint - nor can I use the email since Opsgenie does not confirm the email subscription.

So is there a way to get Heartbeats working from SNS? Alternatively, how do I alert on an integration not receiving anything for a certain period of time?

[Piotr Bzdyl]

Hi @Nick H,

Any update on my issue? Is there any other way to accomplish this? Or a plan in OpsGenie to implement a new feature supporting this?

Best regards,

Piotr