Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,295,777
Community Members
 
Community Events
165
Community Groups

Bitbucket Event Queue Filled up After Custom Plugin Containing Push Event Handler

Hi,

 

I've developed a plugin that checks each repository push and the process of my custom event listener may run long according to the amount of change within the push. Especially during project's automated merge push scheduled job hours I started getting "RejectedExecutionException" due full event queue I also increased event queue size to 9000 with no luck. So, what is the best way to handle custom event listener codes in order not to fill up event queue, I may convert processing to a new job schedule if there is no limit for scheduled jobs ? Simply trying a native java thread also did not worked.

 

-thank you very much

-yusuf

1 comment

Hi,

Code that listens for events needs to complete fast as there are a limited number of threads handling requests on the queue; holding onto one of these threads whilst you do stuff is really bad and not something that will hit you until your system is under load.

I fell foul of this, trying to make HTTPS connections in a bit code listening for events. It caused the bb_alert table to swell to 35Gb, filled up the disk and brought Bitbucket down.

You can configure the queue size with event.dispatcher.queue.size which we set to 9000 and the number of threads through event.dispatcher.max.threads.

The property event.dispatcher.queue.rejectioncooldown has no effect from what I could see. Once your queue fills up Bitbucket starts generating recursively ever more events hence the db table bb_alert growing to 35Gb. Our only way out of the full queue situation was a server restart.

I take my hat off though to the Bitbucket developers as the log file very helpfully not only tells you the queue is full but gives you a stack trace for each thread thats servicing the event queue showing what it is currently doing. From this we easily identified the rogue plugin.

Our fix was to decouple our time consuming code by adding a fifo queue with a scheduled task. If our queue filled up we just discarded new events and logged an error.

Comment

Log in or Sign up to comment
TAGS
Community showcase
Published in Bitbucket

Git push size limits are coming to Bitbucket Cloud starting April 4th, 2022

Beginning on April 4th, we will be implementing push limits. This means that your push cannot be completed if it is over 3.5 GB. If you do attempt to complete a push that is over 3.5 GB, it will fail...

2,170 views 2 9
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you