Heartbeat or other option?

Question

We have devices that send all alerts to the same email address. In this case, I'm working on alerts from Synology, but there are others, like Ubiquiti and many network devices.

I want to set up a heartbeat function for backups. But all alerts go to a single email address.

So is it possible to set up a policy or other feature that will trigger the heartbeat when "backup succeeded" occurs?

I can send an email and have it ignore success and trigger for failure, but I want it to trigger also if it receives no emails at all concerning backup.

Bonus points if I can have it triggered upon the second failure in a row. Sometimes the backup will take longer than a day, so it tries to start one when one is already running.

Russell Johnson_rpe135 · Answer

Thanks for the reply, Andrew.

I'm letting my brain chew on it in the background. Sometimes it does things without me. :o

-Russ

Andrew Laden · Answer

I've looked into this and haven't found a great answer.

Heartbeats in opsgenie are really just to confirm that whatever is sending events to opsgenie is sending them periodically. Ie, you configure your system to send some sort of message every 30 minutes, and tell opsgenie that if it doesn't receive a message every 30 minutes to send an alert. That 30 minute timer resets with every heartbeat message that get sent.

Unfortunately, you cant use the same email for both heartbeats and for regular alert processing.  So you could test that your backup system is working in general by making sure that it send at least 1 message every 25 hours, but you cant use a specific messages to count as the heartbeat. Any message sent resets the timer.

So it really doesn't work for your use case.

Opsgenie is really more of an event processing system as opposed to an event generation system. It has to receive an event to act on. It cant take any action on its own.

So for example, I wanted an alert to get sent if a certain job didn't finish by 4:00 each day. I had to have another system in place to check for that, and send a message to opsgenie if it didnt see the event complete. How to do this really depends on what you are monitoring. For example if your system can generate a file when it completes, you can run a scheduled job (cron or similar) to check if that file exists, and if it doesn't send a mail to opsgenie.

Sorry I don't have a better option for you.