Slack receiving duplicate notifications

[Vivo Bandito]

Hey all, 

I'm having notifications from our slack integration across the board created a duplicate notification when an issue is created. 

Here are some of the troubleshooting steps already taken from my side.

1. I've looked through the slack integration applications>slack and verified that there are not duplicate entries for the slack channels. 
2. I've tested that the same issue is occurring in our production and test environment, both are creating duplicate notifications. 
3. I've turned off the notification and tested if an alert fired. After it was turned off no notifications were created. My thought behind this is that if one did fire after turning off there was either a scriptrunner or post function firing the alert. 
4. I've disable and then re-enabled the slack notification app all together.
5. I've deleted and rebuilt the notification setup. 

as far as I can there there have been no changes to the app via upgrade or upgrades on slack either. Any and all help is appreciated. 

Jira Server for Slack (Official)

Version: 3.0.1

Vendor: Atlassian

Support: Supported by Atlassian

App key: com.atlassian.jira.plugins.jira-slack-server-integration-plugin

[Filipi Lima · Accepted solution]

Hey, folks!

We determined that this was caused by an unannounced change in Slack's REST API response which was not being handled gracefully by the plugin.

To help with this, we released a new version of the Slack for Jira (Official)!

While version 3.0.2 is not yet a permanent fix, it will prevent the duplicate messages from happening.

• This should prevent the duplicate messages but some failures will still be displayed in the logs until we get a permanent fix delivered.

Atlassian Marketplace - Jira Server for Slack (Official) 

Just like its previous version 3.0.1, it is only compatible with Jira version 8.9.1 and newer.

We'll keep adding any future updates on the ticket @Tim Howard kindly provided earlier:

JRASERVER-74410

Cheers!

[Alex Gubar]

Jira Server for Slack (Official) v3.0.2 resolves problem

[Gian Nampi]

Thanks Filipi, Issue fixed after upgrading to 3.0.2.

[Hung Nguyen]

Will there be a fix for JIRA Server 8.8 ? This v3.0.2 seemed to be only for JIRA 8.9 up

[Joe Schofield]

Upgraded to v3.0.2 this morning which fixed this but 3 hours later... it's happening again, any advice please?

[Saravanakumar Balakrishnan]

Hi @Filipi Lima 

I am using slack 4.29 version with jira cloud. but still am receiving duplicate notifications for every changes. it filling my channel threads... 

[Tim Howard]

I just got an answer to my support ticket with Atlassian.  This appears to be a bug as Slack updated their API and the Slack for Jira app doesn't handle the response.

JRASERVER-74410

[Rick Bowen]

Thanks for opening the JRASERVER bug!

Here's the response I received from my support request:

We have been receiving dozen of cases regarding this issue since last Friday and an outage alert has been put in place as we gather all possible information from the specific cases and affected version while development and engineering investigate.

We will keep you informed as the information becomes available.

Looks like they're on it :) 

[Ian Lau]

+1

Started October 13th, 2022 after 5pm EST.

[Isaac Stringer]

Same here +1

[ggodool]

same here.

[Rick Bowen]

If you're having the Duplicate Jira Server Messages in Slack problem and can open a Support Request for Jira Server with Atlassian, I encourage you to open a support request!

https://support.atlassian.com/ 

That will be the best way to get visibility to this problem in order to get it fixed. 

[Mechee]

Same. Started Oct. 11 CDT

JIRA Data Center v8.22.1 and Slack Production 4.28.184 64-bit

[Pam Weber]

We are also seeing duplicate announcements, but only on new integrations defined this week. Integrations that were defined prior to this week are all working as expected.  We are on an older version of the plugin - 2.0.21.  We believe it is a problem on the Slack side, not the Jira side, and have reported it to our partners at Slack.  No response from Slack yet.

[Pam Weber]

Slack responded that we need to report the problem to Atlassian since they are the creator of the integration.  Atlassian will need to work directly with Slack to determine if there is a problem with the integration.  I suggest that everyone on this thread submit a support ticket to Atlassian, citing this conversation.

[Pam Weber]

I submitted a support ticket to Atlassian.

[Pam Weber]

Fixed with 3.0.2 patch.

[jason swan]

Also having this issue across all projects that use slack.  

[Mirko Ellinger]

In the same boat.

[David Press]

Happening here also.

[Oleksandr Li]

Also have the same issue 

[Ravi]

<delete please>

[Florian Bonniec]

Hi @Vivo Bandito 

So when an issue is created, 2 identical notifications are send to the same Slack Channel right ?

What are you using to send slack from JIRA ?

Regards

[Mario Toncer]

Greetings,

same issue as @Vivo Bandito -- started happening today.

On my setup, 2 out of 11 Jira projects are exhibiting such behaviour.

[Ravi]

Same issue, started on the 13th. No changes to config or apps. When I re-added the notification I even got a duplicate message: "<user> 

set Jira notifications from project <project> to appear in this channel."

We're using Atlassian's Slack plugin for Jira Server

Version: 3.0.1

Vendor:  Atlassian

Support: Supported by Atlassian

App key: com.atlassian.jira.plugins.jira-slack-server-integration-plugin

[Florian Bonniec]

I've not review it yet but if you are 3 having the issue starting today without having updated anything I would check on Slack side 😅

[hannah_brooks]

We are also having the same issue, I deleted and rebuilt some of our notifications but this did nothing.

[Matthew R]

Also having the same issue 

[Frederik D.]

Same here.

[Gian Nampi]

We have encountered the same issue using the latest version Jira Server for Slack version 3.0.1. I noticed this behavior a few months ago but it happens randomly, but last Oct 12, 2022 it happens to all subscription.

One thread was updated who encountered the same issue Duplicate messages on Slack when creating tickets.

[Anton Tokarev]

+1

We're experiencing the same behavior.

Using https://marketplace.atlassian.com/apps/1220099/jira-server-for-slack-official?tab=overview&hosting=datacenter plugin.

[Oleksandr Li]

We are also having the same issue

[lnoelstorr]

+1

[Florian Bonniec]

Base on the number of impacted user and the fact that the app did not had any update on JIRA side for a while I would reach out to Slack for this issue.

I'm not familiar with this app but do you have a way to get the webhook that is call on Slack side and try to send something from the command line to see if there is the issue ? If so we can definetly remove JIRA from the equation. 

[Rick Bowen]

Hey @Florian Bonniec

It's not a Slack problem. It's the Jira Server App - https://slack.com/apps/ADZHDM3L1-jira-server - which is an Official, supported Atlassian Slack Application.

I've just now opened a support request with Atlassian and uploaded a Support Zip so that should get the ball rolling internally if no one else has done that yet.

I spent a few hours today parsing through the logs - it's crystal clear that the Jira Server for Slack Plugin - https://marketplace.atlassian.com/apps/1220099/jira-server-for-slack-official - is sending one (and exactly one) Slack message to the Slack App API endpoint, which then results in two (exactly two) messages for each and every event.

This is quite plainly out of the hands of the users, as we do not (and cannot) have any control over the Slack App (which is entirely owned and managed by Atlassian). 

[Ryan Goodwin]

+1

[Alex Gubar]

Jira Server for Slack (Official) v3.0.2 resolves this problem (released 17.10.2022)

[Hylke]

How is this even possible... and how can it be "fixed" by a patch which didn't broke it in the first place!?

We haven't changed anything on the Jira server, nor the plugins in the last few weeks and suddenly we are experiencing this issue as well.

Unfortunately we are currently still at a pre- Jira Server V8.9.1 version of Jira Core (long story don't want to go into), meaning we run slack app V2.0.23 and can't "fix" this, because there is no "fix" in the previous slack plugins?!

I'd like to better understand the cause of the issue and know if there are workarounds?

[Filipi Lima]

Hi, @Hylke,

I'm sorry to hear you're unable to upgrade into a Supported version.

The event that we believe triggered this problem is not on your instance or due to any change under your control.

Apparently sometime last week, the response our plugin gets from Slack changed slightly in format in a way that our plugin could not handle it gracefully, leading to a retry and the duplicate message.

It started sending a "rich_text" element within "blocks" which was not expected.

At this time, we do not have plans to create a fix for an older and unsupported version - Jira 8.9.x itself reached End of Life on May 2022.

[Chad King]

@Filipi Lima   We are seeing this same issue, but are running version 3.0.3.  Is it possible it is not actually fixed?

[Filipi Lima]

Hi, Chad, version 3.0.3 has the permanent fix for the duplicate messages caused by the unexpected elements coming from Slack's response.

I recommend checking if you have multiple subscriptions for the same event and also try a test on a dedicated channel for a test issue/project.

If it remains unclear, you may open a ticket on https://support.atlassian.com/contact/ so we can check it in more details.

Cheers!

Filipi Lima