Jira high disk usage with no user activity

[Joe Prevo]

We are using Jira software 7.6.0 on a Ubuntu 16.4 server. Jira is generating a large amount of disk I/O (averaging 300-500 KB per second).  We only have 3 users that typically use Jira twice a day. There is absolutely no reason for this kind of I/O for an idle system. We have had to move the server's disk off of an SSD to a regular hard drive due to the excessive SQL database I/O.

In investigating it, it appears to be a huge number of inserts and deletes into the rundetails table.  It appears that the culprit might be scheduled jobs for com.atlassian.jira.plugin.ext.bamboo.service.PlanStatusUpdateJob. We have more than a hundred of those jobs and they are scheduled to run every minute. To add insult to injury, if the jobs are really tied to Bamboo, we don't even use Bamboo and have no plan to do so.

Is there any way to stop these jobs or change the interval? We typically only use the system once in the morning and once in the evening. We don't need 100+ jobs running every minute!

Thanks

FYI, while no one was using Jira, I upgraded Jira software to 7.6.1 and now we have 2 more of the jobs under com.atlassian.jira.plugin.ext.bamboo.service.PlanStatusUpdateJob than we had before the upgrade that now run every minute. Do we possible have a problem that jobs are getting created that never go away and they are just building up? If so, is there any way to safely delete the jobs?

[Sue Bridges]

Oh wow, I think this may be my issue that I'm having with heap errors etc.  Like Joe a quiet system, so shouldn't get all of the log file errors I do when I analyse.  I'm currently on 8.1 nothing to do with Bamboo, so sounds like 8.2.2 is the way to go.  Thanks

[Beheerder]

Just chiming in. We are seeing the same problem on a small and quiet Jira Server (7.8.0) on Windows with Microsoft SQL Server, never had any Bamboo integration.

I just deleted 126 rows from table clusteredjob that had key "com.atlassian.jira.plugin.ext.bamboo.service.PlanStatusUpdateJob". This finally stopped the large amounts I/O and huge transaction log backups (500 Mb per day, while the full database backup is only 82 Mb) we were seeing on the server.

The Jira bootstrap log features a line:

[c.a.j.p.e.bamboo.service.PlanStatusUpdateServiceImpl] PlanStatusUpdateJob scheduled to run every 60 seconds

Which seems to have added a new instance on every reboot of Jira, as described in the ticket linked to by Matt Doar.

If I could provide information that would help debugging the issue please let me know. For the moment the mentioned workaround of deleting those rows from the clusteredjob table is satisfactory.

[Shannon S]

Hi Maarten! 

Please feel free to supply any details you have about the issue on the bug ticket. If we need anything further on there our development team will follow up with you!

• JRASERVER-66593

Regards,
Shannon

[Beheerder]

Thanks Shannon, I have now. (Silly me, didn't realize I was allowed to comment on that ticket -- I just assumed it was read-only for me, and did not try.)

[Shannon S]

No worries, Maarten! Thank you for your feedback. :) 

Shannon

[Shannon S]

Hi Joe,

I've created you a ticket in our support system so your Support Team can have a look at your support zip directly.

You should have an email now about that ticket.

Could you please reply to that ticket with a copy of your Support Zip so we can have a look?

Thank you!

Regards,

Shannon

[Matt Doar]

Is this issue something filed for this question: 

https://jira.atlassian.com/browse/JRASERVER-66593

[Shannon S]

Hi Matt!

That bug was raised by the support engineer who looked at the case I made for Joe.

Thank you for checking in!

Regards,

Shannon

[Matt Doar]

My suspicion is that the many jobs could be due to the time on some server not being set as expected. Is ntp working? What timezones are involved?

[Joe Prevo]

Just Eastern Time; Jira calls it New York of course. NTP is working and all our systems (including the server Jira is running on) synchronize to our firewall's NTP service. Checked and they're all within a second of each other. Also, time issues would be very noticeable to us because a product we develop uses SAML authentication that is configured to fail with more than a 5 second drift between system clocks to meet one of our customer's requirements.

The fishy thing to me is that the Bamboo job count goes up by one with each restart of Jira. I never see the count increase if Jira runs without interruption. And the count never decreases unless I delete the jobs out of Clustered Job table.

[Nic Brough -Adaptavist-]

This is well worth checking. 

It's something that bites me a lot when I'm doing SSL stuff, but I forget it might cause issues in other areas!

[Nic Brough -Adaptavist-]

I think you need to track down where all this Bamboo stuff came from.

A new, empty Jira I created this week does not do this.  When I imported a shedload of data in it by importing, it does not do this.  It is now linked to a Bamboo instance with a small handful of new plans created this week, and it does not do this.

You're absolutely right in saying "There is absolutely no reason for this ..." and the ... meaning not just disk I/O, but the database access and jobs.

I think the question needs to be why your system has all of this stuff - what is adding it?  What are they doing?  How did you get it all added?

[Joe Prevo]

I agree. We've been using Jira for years now and have just kept updating through the versions; nothing special. We use very little Jira functionality and there are only 3 users. We only do basic issue tracking, no integration with anything else, no add-ons, etc.

[Joe Prevo]

BTW, we haven't done anything with Bamboo. Never touched it. It got added to Jira as non-removable modules a while back. We'd like to get rid of it but can't. It's a shame that Atlassian keeps adding stuff that you can't get rid of to the base product. It really shouldn't be an Add-on if I can't subtract it off!

[Nic Brough -Adaptavist-]

Thing is, if you've not configured anything to do with it, Jira won't have any way to trigger doing stuff.

I suspect one of your users has done something you don't know about.

Start with a look at

• Application links
• Bamboo configuration

Make sure there's nothing there for either of them!

[Joe Prevo]

It's not that. We have no application links for anything at all and Bamboo configuration shows there is nothing there. I'm the only one with Admin privs so if someone did it, it would have to be me. 

[Nic Brough -Adaptavist-]

So.  If there's no configuration for Bamboo, how do you know it's the Bamboo service (which won't actually do anything unless you've configured it to do so)

What are these hundreds of jobs? 

There is a configuration in your system doing this somewhere, but I can't work out what it is from what you've given us.  You need to work out what is running and how your admins got it there.

Next thing I'd do is go over the add-ons you have.  If any of them mention Bamboo (if you're convinced that's the problem), then disable them.

[Joe Prevo]

Here's a screen capture and you can see the 105 jobs scheduled under the Bamboo group. For a month, we've been at 103 jobs. I just upgraded from 7.6.0 to 7.6.1 and we suddenly jumped to 105 Bamboo jobs that run every minute.

I am and have always been the only Admin and I have never done anything with Bamboo so I have no idea why they are there or why they are killing our system. I can use MySQL tools and see that these jobs just keep running one after the other.

As of now, I just want to know how to safely kill them. I will gladly delete them from the database using SQL if I could figure out what to do. 

[Nic Brough -Adaptavist-]

Ok, I've had a bit more of a dig, and I suspect this is a red-herring. 

This does indeed exist off-the-shelf, but it does NOT cause any disk i/o or database activity in any of the Jira systems I'm looking at.  When the users are quiet, those systems are ticking over doing virtually nothing while having even larger numbers of Bamboo jobs than you do.  Including two that have never been near a Bamboo instance.

I think we're looking in the wrong place.  You've got something else configured to thrash it.

Do your logs have any warnings in them?  Have you tried turning on profiling and maybe SQL logging for a few minutes while it's thrashing, and seeing what they might tell you?

[Joe Prevo]

OK, I've done everything I can possibly think of. The issue is the Bamboo jobs. In doing all this debugging though I did realize something. The number of jobs for com.atlassian.jira.plugin.ext.bamboo.service.PlanStatusUpdateJob increases by one every time I stop and start the Jira service not as a part of the upgrade as I thought.

Out of desperation, I backed up the database and deleted all the clustered jobs for the Bamboo plug and the system looked great. I've started Jira 3 times since I deleted the jobs and I now have 3 Bamboo jobs. With each restart, I get another Bamboo job and the I/O slowly climbs. Sooner or later I'll be back to the same problem unless I keep deleting the jobs.

So how do I get Jira to stop generating a new Bambo job on every restart. Better yet, can I disable Bamboo completely. We don't need it and don't want it.

[Nic Brough -Adaptavist-]

I'm a bit stuck, because this is not happening in the systems I'm looking at.  That tells us someone has configured something in your system to do it.

Yes, there are increasing numbers of lines in that table in my other systems, but there's no pattern of increasing load or anything else causing a problem.   That makes me think your problem is nothing to do with these, but to do with something else.

[Joe Prevo]

Well, I'm at a loss because I am the one and only person that has ever administered this system so if anyone did it, it would have to be me and as far as I know I have never done anything Bamboo-related in Jira.

As for the system load, I didn't notice it until I started getting upwards of 100+ Bamboo jobs. Since they all run once per minute, that's more than 1 per second running on average at that point.

I feel at this point there has to be a bug in Jira because I get a new Bamboo job scheduled every time I restart and that just can't be right. Since we are running the community version there appears to be no way to report this. I guess we'll have to wait for a bigger customer to encounter this and report it. In the mean time, I'll just keep deleting them from the Cluster Jobs table every so often.

[Nic Brough -Adaptavist-]

Since we're both stuck, I'm going to poke Atlassian and see what happens. 

I can't understand why your load goes up apparently in correlation with these jobs when other Jira systems add the jobs but do not.

[Dave Theodore [Coyote Creek Consulting]]

You can't disable the Bamboo plugin, but you can disable all the modules for it.  That would be worth a try.

[Joe Prevo]

Nic, a single job isn't a problem or even a few. It's when hundreds accumulate and run every minute. There's got to be a problem that every Jira restart generates a new Bamboo job that never goes away.

Dave, I was very hopeful. I disabled all the Bamboo modules but a restart still created a new Bamboo job. I also noticed that 7.6.2 was out. Upgraded, which unfortunately gave me yet another Bamboo job! My Bamboo job count is already backup up to 27 just from working on it today after deleting all the original jobs this morning! I think Bamboo was the right name for the product. I've heard how hard it is to get rid of real bamboo!

[Nic Brough -Adaptavist-]

I can see why a few jobs aren't a problem.  What worries me is how you're getting them running, as none of the other Jira systems I've got are running anything.  They do add lines to the table.  But they don't do anything.  I suspect something else is going on.  Just because there's a line in a database table doesn't mean there is a process running and chewing up resource.

At this point, I'd be interested in a thread dump of your system when it's running slowly.  That would tell us what code is being executed, and my instincts are saying we won't find anything Bamboo related.

As for real bamboo - either get a Panda, or bring it round here, I can kill most types of house plant, including bamboo, cactus and aspidistra, just by living with them apparently.

[Pavel Dvořák]

There's no such thing as doing virtually nothing. It means a lot of queries to the database per minute!

Why the jobs are enabled when the plugin is disabled? It doesn't make any sense.

That's why jira is getting slower and slower...