It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage
Highlighted

Bitbucket Event Queue Filled up After Custom Plugin Containing Push Event Handler

Hi,

 

I've developed a plugin that checks each repository push and the process of my custom event listener may run long according to the amount of change within the push. Especially during project's automated merge push scheduled job hours I started getting "RejectedExecutionException" due full event queue I also increased event queue size to 9000 with no luck. So, what is the best way to handle custom event listener codes in order not to fill up event queue, I may convert processing to a new job schedule if there is no limit for scheduled jobs ? Simply trying a native java thread also did not worked.

 

-thank you very much

-yusuf

1 comment

Hi,

Code that listens for events needs to complete fast as there are a limited number of threads handling requests on the queue; holding onto one of these threads whilst you do stuff is really bad and not something that will hit you until your system is under load.

I fell foul of this, trying to make HTTPS connections in a bit code listening for events. It caused the bb_alert table to swell to 35Gb, filled up the disk and brought Bitbucket down.

You can configure the queue size with event.dispatcher.queue.size which we set to 9000 and the number of threads through event.dispatcher.max.threads.

The property event.dispatcher.queue.rejectioncooldown has no effect from what I could see. Once your queue fills up Bitbucket starts generating recursively ever more events hence the db table bb_alert growing to 35Gb. Our only way out of the full queue situation was a server restart.

I take my hat off though to the Bitbucket developers as the log file very helpfully not only tells you the queue is full but gives you a stack trace for each thread thats servicing the event queue showing what it is currently doing. From this we easily identified the rogue plugin.

Our fix was to decouple our time consuming code by adding a fifo queue with a scheduled task. If our queue filled up we just discarded new events and logged an error.

Comment

Log in or Sign up to comment
TAGS
Community showcase
Published in Bitbucket

Powering DevOps with Bitbucket Server & Data Center

Hi everyone, The Cloud team recently announced 12 new DevOps features that help developers ship better code, faster   ! While we’re all excited about the new improvements to Bitbucket ...

1,527 views 0 6
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you