Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,463,724
Community Members
 
Community Events
176
Community Groups

Bitbucket Event Queue Filled up After Custom Plugin Containing Push Event Handler

Hi,

 

I've developed a plugin that checks each repository push and the process of my custom event listener may run long according to the amount of change within the push. Especially during project's automated merge push scheduled job hours I started getting "RejectedExecutionException" due full event queue I also increased event queue size to 9000 with no luck. So, what is the best way to handle custom event listener codes in order not to fill up event queue, I may convert processing to a new job schedule if there is no limit for scheduled jobs ? Simply trying a native java thread also did not worked.

 

-thank you very much

-yusuf

1 comment

Hi,

Code that listens for events needs to complete fast as there are a limited number of threads handling requests on the queue; holding onto one of these threads whilst you do stuff is really bad and not something that will hit you until your system is under load.

I fell foul of this, trying to make HTTPS connections in a bit code listening for events. It caused the bb_alert table to swell to 35Gb, filled up the disk and brought Bitbucket down.

You can configure the queue size with event.dispatcher.queue.size which we set to 9000 and the number of threads through event.dispatcher.max.threads.

The property event.dispatcher.queue.rejectioncooldown has no effect from what I could see. Once your queue fills up Bitbucket starts generating recursively ever more events hence the db table bb_alert growing to 35Gb. Our only way out of the full queue situation was a server restart.

I take my hat off though to the Bitbucket developers as the log file very helpfully not only tells you the queue is full but gives you a stack trace for each thread thats servicing the event queue showing what it is currently doing. From this we easily identified the rogue plugin.

Our fix was to decouple our time consuming code by adding a fifo queue with a scheduled task. If our queue filled up we just discarded new events and logged an error.

Comment

Log in or Sign up to comment
TAGS

Atlassian Community Events