JIRA and Confluence behaving strangely after server reboot (especiall add-ons page)

[Alastair McBain]

I recently rebooted my server to run on a newer kernel that had a fix for the Dirty COW vulnerability. I forgot an argument and so the shutdown didn't go to cold and I failed to pick up the new kernel and had to try again shortly after the system came back up again.

When the server was finally ready to go the second time I logged in to JIRA and Confluence to see if they were alright. The JIRA system dashboard I'd set up didn't show until I refreshed a bunch of times, and I tended to get entirely blank (white) or solid gray pages. I restarted JIRA and was able to see the dashboard but the add-on manager (UPM?) won't list my installed add-ons. The list is blank.

Using Firebug I was able to determine the page has this error:

Error: undefined missing ManageAddonFlows
}else{throw new Error(b+" missing "+h)

/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/en_UK7zundm/71003/b6b48b2829824b869586ac216d119363/2.21.4/_/download/batch/com.atlassian.upm.atlassian-universal-plugin-manager-plugin:upm-requirejs/com.atlassian.upm.atlassian-universal-plugin-manager-plugin:upm-requirejs.js (line 4764, col 13)

The server log is also full of messages about a "broken pipe" but I can't see any of the connections it claims my browser is killing (the usual reason for this message). By full, I mean JIRA scribbles all over the log. Whatever causes this it does it a lot. Prior to this I was seeing JS errors potentially indicating that files were not complete before an attempted execution.

I was able to determine that there were updates before things started going south, so I tried updating the UPM plugin manually (according to the process on the Atlassian wiki) as that was one listed as having an update. Unfortunately this didn't seem to have any effect and caused some browsers to see these two messages:

This version of JIRA is unrecognized to the Atlassian Marketplace. It might be a non-standard build or be too new to appear.
All add-on data below is for the latest known product version. Add-on compatibility with this JIRA version is unverified; install add-ons at your own risk.

This version of JIRA is unrecognized by the Atlassian Marketplace. It might be a non-standard build or be too new to appear. Visit the Atlassian Marketplace website to continue your search.

Confluence for the most part seems to be fine, however the add-ons page there had issues for a while that magically resolved themselves after a restart of the app. I still see this error in Firebug's console, but I don't really know what to do about it.

TypeError: h is null
if(!upm.onDemand){AJS.$.ajax({url:upm.rootNotificationsUrl,type:"get",dataType:"...

/confluence/s/6c0981174e94142364564819dc0d1413-CDN/en_GB/6212/d125ddfe4e16e78d1fea8ef42a18979f09319385.1/e412b1cd17ce503b35d4757b5b408d9b/_/download/contextbatch/js/admin,atl.admin/batch.js?cache=false&locale=en-GB (line 141, col 1)

I am running JIRA version 7.1.0 build 71003, and had the plugin manager version 2.21.1 and updated it to 2.21.4. Confluence is version 5.9.5 build 6212 and says it's running the latest of all installed plugins. My system is a Debian Wheezy (yes, I know, that's not exactly the latest, but things I think were working fine until after the reboot.)

Long story short, has anyone else seen these errors before or encountered these problems and know how to solve them? Is this something I need to reinstall JIRA and Confluence over to fix?

[Jonas Andersson]

Was only the kernel upgraded or did you do an apt-get update/upgrade too? One thing i would check it that JIRA is started with the correct Java version. In case of a system upgrade, this might have changed which can be reflected in many different way. On my testing environment i am not shy to use kill -9's whenever i see fit, i accidentally killed a whole bunch of processes including the local mysql server once, and even doe ungracefully was stretched, JIRA did come back up. A think i noticed with the new JIRA is that it sometimes takes quite some time to start, in some cases a minute or more after i see the "JIRA is now running on blabla:8080", so be patient. UPM in particular is super sensitive to if you are accessing it through the real application url, FQDN or server alias, so make sure you are hitting it on the right URL. Lastly, if you have a mod proxy infront of JIRA it makes sense restarting it again. Ive noticed that if JIRA is down for a longer time, apache can exhaust it self, and prespawned threads can sometimes give misleading redirections and results. If you are running JIRA behind a mod proxy, try to reach it on its native port and see if you see the same things. As for the manual UPM installation, make sure you only have one UPM jar / directory (normally in installed-plugins, .osgi-plugins). Do a "locate jar | grep universa" and make sure you don't see a whole bunch of versions floating around. Also, if the UPM version ytou had before worked, it most likely will work now, so if you have a copy of it, put it back again. You can upgrade it via the addons later. And yes, i don't like to spend my bandwidth on linebreaks

[Alastair McBain]

It's using its own built-in Java that it shipped with, which is 1.8. These issues persist well after startup. I did try restarting Apache since it was behind a reverse proxy, but it seems to be the same as before. It was working some time before so I know I have all the bits with the proxy worked out so that JIRA didn't get confused.

Edit to clarify: I verified the Java version back when I posted this by viewing the system stats/info page.