How to temporaly silent opsgenie notifications?

[Daniel Leite]

With the recent cloudflare network problem, our sites went down. Of course, opsgenie received alerts form several systems and kept beeping and showing up notifications  for all of then.

So the question is how to silent all notifications for X minutes, when we know already about the problem or a service is flapping.

They should still show in the site/app, but just do not show notification nor alerts... and of course, after the X minuted, any new alert should send notification again

Thanks for the help

[Nick H · Accepted solution]

Hi Daniel,

Thanks for reaching out. I think your best bet for this scenario would be a Maintenance Policy which allows you to enable or disable multiple policies/integrations with a single entry during a specified window.

If you would like the alerts created in your Opsgenie account, but notifications delayed/suppressed - you'll want this Maintenance window to enable a Notification Policy. If you would like the alerts to NOT be created you will want to disable an integration - obviously if an alert isn't created, no user(s) will be notified.

Both the Notification and Maintenance policies can be set on the team level under the Policies tab. If you were to lean towards enabling/disabling a Notification policy, you will first want to create this.

Within the filters of the Notification policy - you can select ALL alerts to be delayed/suppressed, or get more granular with the conditions only looking for cloudflare alerts:

Once this Notification policy is configured, you will now be able to search for and enable it during the Maintenance window. Within a Maintenance policy, specify the timeframe  and the Notification policy.

If you were to lean the other route of either enabling/disabling an integration and avoid any alert creation during this window - you'd simply want to adjust the filter of the Maintenance policy:

You can also select the + Add rule button in the Maintenance policy to enable or disable multiple policies/integrations.

Best,

Nick

[Daniel Leite]

While not very friendly way to do this, this do work!

it is not very friendly because  it is way too complex and requires multiple steps, most of our team do not know opsgenie well enough to even know how this setting work. But the worse is that requires a user with enough permissions to build and enable those settings, so at least it requires a escalation to the next level to be able to create those emergency maintenance settings.

A better option would be a user "silent/auto-ack all for..."  x minutes, next to the "ack all, close all"

It would be something that would not be used normally, but very useful to silent a alert spike, giving the team time to work in the problem and not in the never ending alert flood

maybe i need to create a feature request... (but how to do it ! ) :)

[Nick H]

Hi Daniel - thanks for the feedback. You can alternatively suppress the notifications of an integration (found under Settings -> Configured integrations OR a team's Integration tab) - but will need to remember to turn this option back on after the fact:

[Carlos Reyes]

Is there a way to have a recurring Maintenance window?

We have some nightly tasks that generate alerts.. 

I cant find a way to have a nightly maintenance window say 21:00 to 21:15 

Any tips?

[Nick H]

Unfortunately not at this time, but we do have an open feature request for this that I have added you to. Our tickets are not public, so I'll reach out directly with any updates/developments on the ticket. And if you'd like to contact support for an update yourself, the ticket for reference is ALX-818; Add recurring time windows as an option for Maintenance.

[Thomas M.]

+1 for scheduled maintenance windows. Please include me on the updates.

[Nick H]

Will do! No updates on this ticket yet, but I've added you to the ticket in case anything develops.

[Nic.Zanelli]

Can I suppress notifications of an integration for only 1 entity?  We have 10 products that use the 1 integration - central monitoring tool - and we split them out by entity.  In a maintenance window scenario, I want to not have alerts created for 1 of the 10 products (entities).  I don't want to turn off alert creation for the other 9 products.

I can see how to suppress notifications by entity and I can see how to suppress alerts by integration, but I want to suppress alerts by entity.

[Nick H]

Hi Nic,

You'd want to configure a Notification Policy that suppresses the notifications for this entity/types of alerts.

Within the policy, filter on the field that the entity is parsed through (most likely the Entity) -> then have the Maintenance window enable this policy when either scheduled or started:

[Kevin Six]

Please add me to the feature for planned recurring maintenance windows.

[JiChung]

+1 please, as we would also like to suppress alerts for a specific service until certain time every day.

[Christopher Emery]

+1 please add me to the recurring scheduled maintenance window feature request. This will be hugely beneficial to our usage of OpsGenie.

[Darren Krohn]

@Nick H What is the status of this recurring scheduled maintenance item?  Will you please add me to the ticket if it helps as well? 

In my opinion, it'd be nice to see the Maintenance Policy feature allow you to enable/disable a Action Policy that is set up as well (it only lets you choose to Enable/Disable Notification Policies or Integrations).   For example, we have an Opsgenie  Action Policy that automatically tries to start back up an integrated tech item whenever Opsgenie receives an alert that our tech item is down.  During our maintenance windows, we purposely want the alerts to still occur, but we purposefully want to keep the tech item shut off for the multi-hour maintenance window, and we do not want our Action Policy to trigger.  I have made ways to work around this, but it'd be helpful to have.

[Nick H]

Hi @Darren Krohn ,

No update as of now on the recurring scheduled maintenance feature request. If/when we have an update, we'll reach out here! 

[Bruno Araújo]

Is it possible to achieve this without using policies?

[Nick McNeal]

Wondering if this is the same issue my team has been trying to iron out for the last few weeks- we have an email integration that creates alerts for incoming emails from QRadar for one of our clients that we want to suppress during hours that we do not provide support for the client. Here's the notification policy that we created under the team that gets those alerts. Problem is, it doesn't appear to have any effect at all:

[Nick H]

Hi @Nick McNeal ,

If this notification policy is consistent, and should always apply M-F 07:30-16:30, then I don't think enabling this via a maintenance policy is needed. You should be able to simply just keep the notification policy enabled at all times.

Regardless, what exactly is not working as expected? Is the notification policy not silencing these alerts during those days / hours? You may want to check your regex is correct as well. I typically use something like this: https://regex101.com/

It's also important to note that the hours defined in notification policies are applied to the account's timezone. So if your personal timezone varies, then that might be an issue: https://support.atlassian.com/opsgenie/docs/time-constraints-and-timezones/

Hope that helps. 

[Nick H]

Feel free to also open a ticket with our support team here: https://support.atlassian.com/opsgenie/

I noticed you blurred some of the above screenshot so if you needed to keep this private, we can take a look at things behind the scenes.

[Nick McNeal]

That was quick! Yeah, I originally had it set up without the maintenance policy- just getting desperate and trying different things at this point. The regex should work- it's using the same exact phrase that we use for the integration, which works fine (it's just the name of our client-hence the blurring). Basically, the problem is that we're still getting alerts during those "alert suppression" hours for alerts with those tags. The alerts are never suppressed during any hours for any time zone. It's like it just doesn't work. I have the right team, the right tag, the right hours, etc., and we're still getting alerts all hours of the day. 

[Nick McNeal]

I've tried tags contains ClientName, as well. FWIW

[Nick McNeal]

Thanks for the help, Nick! I've opened a support request, so that we can take a look at the issue behind the scenes with your support team. There's probably something I've overlooked in the setup, and another set of eyes should clear things up. 

[Nick H]

Hi @Nick McNeal ,

Following up here quick to share some insight into your use case with the Community. 

Basically Nick wanted to suppress notifications AND outgoing requests during off-hours for their Jira instance - so no issues are created during certain hours.

Notification policies can only suppress notifications sent to users, and cannot stop outgoing requests to Jira, or other integrations.

As a workaround we configured an alert policy under the team receiving these alerts that was restricted to time intervals, and was stripping the dynamic responder field from the alert - which essentially removes any responders from the alert entirely:

This keeps the team receiving the alerts as the Owner team on the alert, but removes them as a responder. That stops notifications from being sent to the team, and outgoing requests to other integrations:

[Ezra Van Den Bosch]

Why can't it be more user friendly? each change we make affects other alerts and vice versa

[Ezra Van Den Bosch]

We only want to disable an integration for the specific alert - so alerts don't go through to MS Teams for instance. 

We don't want to disable the complete integration for a set period of time.

[Ezra Van Den Bosch]

I am trying it with the alert policy now removing the dynamic responder field - will see if that works as expected

[Ezra Van Den Bosch]

For my use case the following seems to fit:
1) alert policy that removes the responder field based on condition
2) notification policy that auto-closes the ticket based on the same condition

In this case the alert seems to be completely "silenced"

[Ezra Van Den Bosch]

Only downside is that we do still receive an email when the ticket is closed

[Ezra Van Den Bosch]

no email when it's set to auto-close though so that's good news!