JIRA Rebuilding Index Stuck - Restart Gives 'Jira Locked Error'

[Mark]

Hi, we have been preparing our Jira Index for upgrade and updating the plugins accordingly.

When rebuilding the instance after upgrading a plugin, it became stuck and hung. To resolve this, we tried to restart the JIRA application and now we are stick with the 'Jira locked error'

We have tried the following:

• Deleted '.bundled_plugins' & '.osgi_plugins'
• Increased the memory limit
• Given the plugins more time at startup (increased to 300 then 600)

But none have resolved the issue.

Any help would be appreciated.

Thanks!

[Mark]

Sorted. I misunderstood my admins comments about what he tried on the above documentation. For information, the issue was resolved by re-enabling the plugins in the DB. Thanks very much for the help Dave.

[Dave Bosman [Realdolmen]]

Hi Mark, 

I'm glad that you where able to solve it. 

Best of luck with your upgrade. Don't forget to create a backup first

PS: For future reference, also make a backup before upgrading your plugins

Regards

Dave

[Mark]

Yes, I certainly will! Thanks again Dave.

[Mark]

For current context, we have now tried everything on the documentation:

https://confluence.atlassian.com/jirakb/troubleshooting-jira-startup-failed-error-394464512.html?_ga=2.3729273.814050296.1535055278-790362204.1520109706

[Dave Bosman [Realdolmen]]

Hi Mark,

You could try starting in safe mode if you think it is caused by one of the plugins. 

 *edit*

Is there an antivirus running on the server? 

According to Atlassian:

JIRA applications and anti-virus software are known to be mortal enemies - It's hard to have both of them at the same place at the same time. During startup, JIRA applications create quite a number of temporary files that it has to read from. If an anti-virus software goes through all of  these files (which are usually in the JIRA_INSTALLATION or the JIRA_HOME directory), it can hinder or stop the startup process from proceeding normally. 

Exclude your JIRA INSTALLATION DIRECTORY and your JIRA HOME DIRECTORY from being scanned by the anti-virus. However, sometimes, even though an exception has been added, the problem might still persists, in those cases, please try disabling your anti-virus entirely.

Regards,

Dave

[Mark]

Hi Dave, will try safe mode, thanks. No there is no AV on the server.

[Dave Bosman [Realdolmen]]

Hi Mark,

If that does not work, can you please check the log files?

you can also check this  to unlock. 

Regards

[Mark]

Hi Dave, thanks for this. We have unlocked now so trying that. Where would we find the log files? A silly question, I know but under home we have only access.log

[Dave Bosman [Realdolmen]]

Hi Mark,

If you did not make any modifications where the logs are stored and did not alter the install and home directory you should be able to find different log files in following 2 locations

/var/atlassian/application-data/jira/log  

/opt/atlassian/jira/logs

Regards

Dave

[Mark]

Thanks, Dave, as a side note, our objective here was to update JIRA to the latest version. It is not a major upgrade as the application was only updated last year.

Could you see that performing the upgrade would potentially solve the issue or is it not likely to help/make it worse?

We are trying safe mode first to see if the issue is plugin related, as the last update was made to TEMPO before the index got stuck.

[Dave Bosman [Realdolmen]]

If i where you, i would not perform the upgrade until i solved this issue because if something happens during the upgrade you might end up with 2 issues. And that would make it harder to troubleshoot. 

And besides that it is likely that an upgrade would not solve the current issue. 

[Mark]

Thanks, Dave. I thought as much.

Here are the contents of the log file which relate to plugins, can you see anything in here? We are currently trying safe mode:

'com.infosysta.jira.mobile' - 'Mobile for JIRA' is unaccounted for.

It was loaded from /var/atlassian/application-data/jira/plugins/installed-plugins/plugin.866838187341300444.MobileForJIRA-1.9.4.jar

'com.arsenalesystems.dataplane' - 'Arsenale Dataplane' is unaccounted for.

It was loaded from /var/atlassian/application-data/jira/plugins/installed-plugins/plugin.4035708384647779741.arsenale-dataplane-2.3.4.jar

'com.atlassian.jira.jira-share-plugin' - 'Atlassian JIRA - Plugins - Share Content Component' is unaccounted for.

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-share-plugin-7.3.7.jar

'com.atlassian.jira.jira-core-reports-plugin' - 'Atlassian JIRA - Plugins - Core Reports' is unaccounted for.

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-core-reports-plugin-7.3.7.jar

'com.atlassian.jira.gadgets' - 'Atlassian JIRA - Plugins - Gadgets Plugin' is unaccounted for.

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-gadgets-plugin-7.3.7.jar

'com.atlassian.jira.rest' - 'Atlassian JIRA - Plugins - REST Plugin' is unaccounted for.

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-rest-plugin-7.3.7.jar

'com.okapya.jira.checklist' - 'Checklist' is unaccounted for.

It was loaded from /var/atlassian/application-data/jira/plugins/installed-plugins/plugin.5110310521253732445.checklist-3.1.13.jar

[Mark]

A safe mode start does not resolve it, unfortunately. We still get the same locked error. Any further ideas?

[Dave Bosman [Realdolmen]]

For the moment i am a little puzzled. :-S

Have you already created a support ticket with Atlassian?  

Maybe they have some ideas that i did not think off yet. 

Did you find an atlassian-jira.log or a catalina.out in your log file dirs? 

[Mark]

Hi Dave, yes, us as well! It's not something we have come across before.

Normally we can resolve these issues with a restart or memory increase.

We cannot raise a ticket with Atlassian as our license no longer allows us to. 

We have a catalina.out in the log files. Should I post this?

[Dave Bosman [Realdolmen]]

So, no support anymore? 

You do know that you need a valid and active license to upgrade right?

You can post the log, maybe we can see something in that file. 

[Mark]

In regard to the license question, it is paid and current, we are just unable to raise a ticket as we have only 10 users (its the cheaper one)

[Mark]

a little part from atlassian-jira.log which looks helpful----

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-issue-nav-plugin-8.2.6.jar

'com.atlassian.jira.jira-quick-edit-plugin' - 'Atlassian JIRA - Plugins - Quick Edit Plugin' failed to load.
Application context initialization for 'com.atlassian.jira.jira-quick-edit-plugin' has timed out waiting for (objectClass=com.atlassian.jira.rest.v2.issue.builder.BeanBuilderFactory)

It has the following missing service dependencies :
&beanBuilderFactory of type (objectClass=com.atlassian.jira.rest.v2.issue.builder.BeanBuilderFactory)
&beanBuilderFactory of type (objectClass=com.atlassian.jira.rest.v2.issue.builder.BeanBuilderFactory)

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/jira-quick-edit-plugin-3.2.0.jar

'com.atlassian.jira.jira-issue-nav-components' - 'jira-issue-nav-components' failed to load.
Application context initialization for 'com.atlassian.jira.issue-nav-components' has timed out waiting for (&(objectClass=com.atlassian.jira.rest.v2.issue.builder.BeanBuilderFactory)(objectClass=com.atlassian.jira.rest.v2.issue.builder.BeanBuilderFactory))

[Dave Bosman [Realdolmen]]

This is not really helpful as it only states that it failed to load/timed out but it does not give any insights into the real reason unfortunately. 

Can you check what java version you are using on the server? 

And i presume the server is powerful enough to handle the JIRA environment? 

Did you make a backup before making the modifications? 

Because worst case scenario - Install jira from scratch and import the backup - is sounding like a very good option to me right now

[Mark]

Hi Dave, thanks! We are checking on the JAVA version now as that was on our list of things to try. 

Server is fine to handle it. We have tons of free resource.

We do have a backup yes, but I made the mistake of intending to back up following the upgrade of plugins, so our back up is from a few weeks ago.

Not ideal, but a solution.

Thanks very much for your help with this one so far.

[Mark]

The JAVA version looks like version 8.

Do you know how I can post my log files on here? I have no attachment option.