Bitbucket Event Queue Filled up After Custom Plugin Containing Push Event Handler

[Yusuf TAMBAG]

Hi,

I've developed a plugin that checks each repository push and the process of my custom event listener may run long according to the amount of change within the push. Especially during project's automated merge push scheduled job hours I started getting "RejectedExecutionException" due full event queue I also increased event queue size to 9000 with no luck. So, what is the best way to handle custom event listener codes in order not to fill up event queue, I may convert processing to a new job schedule if there is no limit for scheduled jobs ? Simply trying a native java thread also did not worked.

-thank you very much

-yusuf

[Paul]

Hi,

Code that listens for events needs to complete fast as there are a limited number of threads handling requests on the queue; holding onto one of these threads whilst you do stuff is really bad and not something that will hit you until your system is under load.

I fell foul of this, trying to make HTTPS connections in a bit code listening for events. It caused the bb_alert table to swell to 35Gb, filled up the disk and brought Bitbucket down.

You can configure the queue size with event.dispatcher.queue.size which we set to 9000 and the number of threads through event.dispatcher.max.threads.

The property event.dispatcher.queue.rejectioncooldown has no effect from what I could see. Once your queue fills up Bitbucket starts generating recursively ever more events hence the db table bb_alert growing to 35Gb. Our only way out of the full queue situation was a server restart.

I take my hat off though to the Bitbucket developers as the log file very helpfully not only tells you the queue is full but gives you a stack trace for each thread thats servicing the event queue showing what it is currently doing. From this we easily identified the rogue plugin.

Our fix was to decouple our time consuming code by adding a fifo queue with a scheduled task. If our queue filled up we just discarded new events and logged an error.