Heartbeats Alerts trigger even when pings are successful

[Sid]

Title.

There are emailed alerts triggering on multiple servers indicating a heartbeat has not been received.

I checked the history of each server's scheduled runs to ping a heartbeat and they all show success well within the time limit before an alert is supposed to trigger.

If I go to the OpsGenie heartbeats page, all of the servers that are supposedly 'off' are all showing active. These servers are not turned off or restarted for maintenance either on my company's end.

I found this happens every Tuesday between 2:00 AM and 2:06 AM Central Standard Time. The 'last received at' will all say roughly 11:50 PM, but this is incorrect as stated earlier as several heartbeats for each server successfully ran between 11:50PM to 2:00 AM. 

Is there a weekly maintenance from OpsGenie that causes this phenomenon? Or is there another place to check?

[Sid · Accepted solution]

Thank you Agaci for pointing me there. There were no global policies defined for my account unfortunately.

Through support ticket, we narrowed down that it's a confusion of time zone difference of the account and my profile. It appears a 5 minute padding of pinging may not be totally foolproof so I will be increasing the rate of pings.

[Agaci Avinas]

Hi Sid,

I am Agaci from Opsgenie support. Happy to help you today.

I understand that heartbeat pings are not reaching Opsgenie during a specific time every week. 

To answer you question, we do have maintenance policy in Opsgenie which you could check under Settings --> Global Policies. However, maintenance policies will not be able to disable Heartbeats.

In this case, we would like to check detailed logs to find if Opsgenie has received the ping. Could you please reach us by creating a ticket in this link. We would like to get access to the instance and check the logs to narrow down the issue. 

Regards,

Agaci 😃