ScriptRunner listeners missing events triggered by a Post Function

Marc-Antoine Carrier

Hi, I have encountered an issue with Jira data center v.9.2.0 and ScriptRunner v.6.44 where some Listeners are missing Post Function events when Jira is busy dealing with multiple different calls. To put you in context, here is what I am currently working on:

I am working on creating integration tests between a web API and Jira. The web API's job is to create ScriptRunner Listeners in the Jira instance its pointing to. Each custom ScriptRunner Listener will listen for either a Generic System Event or a Custom Event triggered by a Post Function when a particular type of issue is transitioned to specific statuses. The integration tests were created to make sure the Listeners' logic created by the web API's is always free of regressions in Jira. Currently, we have around 400 different tests that make sure our logic is always right once it gets on Jira. Each single test will create between 3 to 12 Listeners (i.e.: between 1200 and 4800 listeners are created during the test suite lifetime) and will create the conditions for the different Listeners to be triggered by the Jira event system. Since multiple listeners can be listening for the same event, we added specific checks in our scripts to make sure the right issues trigger the right listeners (e.g.: Does the component fit what the script is looking for?). To speed things up, the tests are run in parallel with a maximum of 8 tests running concurrently. Since we can't decide when certain actions take place, the following actions could occur concurrently during the lifetime of the test suite:

• Listener creation/deletion (retries were added to prevent any concurrency problem)
• Issue creation/deletion
• Component creation/deletion
• Post function event triggered by the transition of an issue

Issue:

Recently, I have been troubleshooting a flakiness issue with our integration tests where around 10 random tests are failing on every test run for reasons like issues were not created properly by our scripts or some issues were not transitioned correctly by the scripts. Re-running the failed tests after the test run will have them turn green again.

Troubleshooting:

To understand the problem I was facing, I added logs to our different scripts to find out if certain issues were tossed out for the wrong reason… that was not the case. The listeners that were not looking for that event, properly tossed out the event. However, what I found was that the listener that should have been triggered by the event, did not even get to see the event. I first thought maybe that listener was not properly created, but I looked at the Audit logs and it was indeed properly created.

Questions:

I would like to know:

1. Have you ever seen a problem of lost post function event in Jira? If so, how did you troubleshoot it?
2. Do you have any tips on how to troubleshoot the Jira event system?
3. Are there any hidden configuration items related to the Jira event system?
4. Knowing that ScriptRunner has some concurrency issues, could this lost event issue be related to any Scriptrunner concurrency problem?

Marc-Antoine Carrier

For visibility, I created a ticket with the Adaptavist support and we ended figuring out what was happening. When you create a new Listener on Jira, all previously registered Listeners will first be un-registered (all of them) and then they will be re-registered with the newly created Listener. If an event occurs during the time gap between the un-registering and the re-registration of the Listeners, then this event will be "lost" because no Listeners were registered to listen to it.

This gap is usually very small and will probably not impact most use cases on a production instance. However, in my case, I was creating and deleting Listeners at the same time events were occurring on my local test instance. That is why I was missing events. According to my discussion with support, a good way to prevent that would be to synchronize the creation/deletion of Listeners and let the events occur when no operations are happening on the Listeners. I have been working on that for some time and it seems to be working so far.

Something to keep in mind: creating/updating and deleting Listeners are not thread-safe operations according to this ticket SRPLAT-1730. I have been implementing retries to circumvent that limitation and that has been working for me so far.