Error Upgrading Confluence from 7.4.17 to 7.13.0

[Michael Cromer]

Hello,

I am trying to upgrade my on-prem Confluence Data Center environment from v7.4.17 to v7.13.0. I want to go here instead of directly to the latest version because our Postgres database version is v9.6, and I want to also upgrade that to the latest available version.

I have a testing environment where I've stood up another v7.4.17 environment, identically deployed via automated Helm charts into a Rancher Kubernetes cluster. When I attempt to deploy a v7.13.0 Helm chart, the upgrade seems to work, but then it receives an error at some point. After the error, the deployment is in a stuck state and I cannot access the application. This same error occurs when attempting to upgrade either a new deployment, or a deployment with an imported copy of our production data. The error is below.

Any assistance that could be shared in helping me resolve this issue would be greatly appreciated. Thank you.

Type Exception Report

Message Error creating bean with name 'manualBackupsAdminTask' defined in class path resource [applicationContext.xml]: Cannot create inner bean 'com.atlassian.confluence.admin.criteria.BackupsAreManualCriteria#5928f900' of type [com.atlassian.confluence.admin.criteria.BackupsAreManualCriteria] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.confluence.admin.criteria.BackupsAreManualCriteria#5928f900' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'scheduledJobManager' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scheduledJobManager' defined in class path resource [schedulingSubsystemContext.xml]: Cannot resolve reference to bean 'scheduledJobManagerTarget' while setting bean property 'target'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scheduledJobManagerTarget' defined in class path resource [schedulingSubsystemContext.xml]: Cannot resolve reference to bean 'managedScheduledJobRegistry' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedScheduledJobRegistry' defined in class path resource [schedulingSubsystemContext.xml]: Cannot resolve reference to bean 'managedSystemJobs' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedSystemJobs' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot create inner bean 'com.atlassian.confluence.schedule.ManagedScheduledSimpleJob#65d4ae2c' of type [com.atlassian.confluence.schedule.ManagedScheduledSimpleJob] while setting constructor argument with key [21]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.confluence.schedule.ManagedScheduledSimpleJob#65d4ae2c' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot create inner bean 'com.atlassian.confluence.impl.search.IndexFlushScheduledJob#62b8ab81' of type [com.atlassian.confluence.impl.search.IndexFlushScheduledJob] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.confluence.impl.search.IndexFlushScheduledJob#62b8ab81' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot resolve reference to bean 'luceneContentIndexFlushScheduler' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'luceneContentIndexFlushScheduler' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot resolve reference to bean 'luceneContentIndexManager' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'luceneContentIndexManager' defined in class path resource [indexingSubsystemContext.xml]: Cannot resolve reference to bean 'tenantedContentLuceneConnection' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tenantedContentLuceneConnection' defined in class path resource [indexingSubsystemContext.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.bonnie.LuceneConnection]: Constructor threw exception; nested exception is com.atlassian.bonnie.LuceneException: org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /var/atlassian/application-data/confluence/index/write.lock

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'manualBackupsAdminTask' defined in class path resource [applicationContext.xml]: Cannot create inner bean 'com.atlassian.confluence.admin.criteria.BackupsAreManualCriteria#5928f900' of type [com.atlassian.confluence.admin.criteria.BackupsAreManualCriteria] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.confluence.admin.criteria.BackupsAreManualCriteria#5928f900' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'scheduledJobManager' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scheduledJobManager' defined in class path resource [schedulingSubsystemContext.xml]: Cannot resolve reference to bean 'scheduledJobManagerTarget' while setting bean property 'target'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scheduledJobManagerTarget' defined in class path resource [schedulingSubsystemContext.xml]: Cannot resolve reference to bean 'managedScheduledJobRegistry' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedScheduledJobRegistry' defined in class path resource [schedulingSubsystemContext.xml]: Cannot resolve reference to bean 'managedSystemJobs' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedSystemJobs' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot create inner bean 'com.atlassian.confluence.schedule.ManagedScheduledSimpleJob#65d4ae2c' of type [com.atlassian.confluence.schedule.ManagedScheduledSimpleJob] while setting constructor argument with key [21]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.confluence.schedule.ManagedScheduledSimpleJob#65d4ae2c' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot create inner bean 'com.atlassian.confluence.impl.search.IndexFlushScheduledJob#62b8ab81' of type [com.atlassian.confluence.impl.search.IndexFlushScheduledJob] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atlassian.confluence.impl.search.IndexFlushScheduledJob#62b8ab81' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot resolve reference to bean 'luceneContentIndexFlushScheduler' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'luceneContentIndexFlushScheduler' defined in class path resource [schedulingSubsystemManagedContext.xml]: Cannot resolve reference to bean 'luceneContentIndexManager' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'luceneContentIndexManager' defined in class path resource [indexingSubsystemContext.xml]: Cannot resolve reference to bean 'tenantedContentLuceneConnection' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tenantedContentLuceneConnection' defined in class path resource [indexingSubsystemContext.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.bonnie.LuceneConnection]: Constructor threw exception; nested exception is com.atlassian.bonnie.LuceneException: org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /var/atlassian/application-data/confluence/index/write.lock
	

[Michael Cromer · Accepted solution]

I've done several successful deployments now with multiple Helm chart versions of Confluence and Postgresql, and my challenge was actually not waiting long enough for Postgres to fully initialize and be ready before launching my Confluence Helm chart. New browser had nothing to do with it.

[Yevhen]

Hi @Michael Cromer Did you check https://confluence.atlassian.com/confkb/unable-to-start-confluence-or-enable-the-usage-stats-plugin-due-to-cannot-forcefully-unlock-a-nativefslock-error-688884516.html? Looks like exactly the error you see. Permissions issues?

[Michael Cromer]

Hello Yevhen,

Thank you for your response. I apologize for the delay in my reply. I ended up redeploying my kubernetes environment as it was in an unusable state when trying to resolve this challenge.

I did check permissions on this staging 7.13.0 directory and compared it against our production 7.4.17 directory, and the permissions (drwxr-x---) line up between the deployments.

I also monitored the directory where the write.lock file appears, and deleted it as soon as it appeared to see if that would resolve the issue, and it didn't.

One thing to note: my directory that contains the write.lock file was in a different location than stated in the link, and I "stop" the application by reducing the number of Confluence replicas in kubernetes to 0, and then bringing it back up to 1.

Any further assistance is appreciated.

Thank you.

[Yevhen]

Hey Michael! Do I get it right? You scaled sts to 0, deleted lock file, scaled to 1 then to 2 and the issue persists? 

Can both replicas use a shared volume for example? 

/var/atlassian/application-data/

is a local home which is persisted individually for each statefulset replica. Perhaps the two pods share it somehow?

[Michael Cromer]

Good Morning Yevhen,

No, the process you listed isn't what I was doing during my comment. I was logged into the container directly and actively monitoring the directory where the write.lock file was appearing, and I deleted it as soon as I saw it, thinking/hoping that that would resolve the issue.

After reviewing your comment, I gave that a shot: I deployed the helm chart, added a trial license key, and then when the error appeared, I scaled down to 0, removed the write.locl file, and then scaled back up to 1. I now get what appears to be a general system error, pasted below. Scaling up to 2 does not change the end result.

For what it's worth, we generally only run Confluence with one active replica, so I don't know that this would necessarily be the issue.

Thank you for your response, and I look forward to your next reply.

System Error

A system error has occurred — our apologies!

For immediate troubleshooting, consult our knowledge base for a solution.

If you would like to receive support from Atlassian's support team, ask your Confluence administrator to create a support issue on Atlassian's support system with the following information:

1. a description of your problem and what you were doing at the time it occurred
2. a copy of the error and system information found below
3. a copy of the application logs (if possible).

Your Confluence administrator can use the support request form to create a support ticket which will include this information.

We will respond as promptly as possible.
Thank you!

Return to site homepage…

The SystemInformationService could not be retrieved from the container. Therefore very limited information is available in this error report.
The SystemInformationService could not be retrieved due to the following error: java.lang.IllegalStateException: Spring Application context has not been set

Cause

com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Spring Application context has not been set
    at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)

caused by: java.lang.IllegalStateException: Spring Application context has not been set
    at com.atlassian.spring.container.SpringContainerContext.getComponent(SpringContainerContext.java:48)

Stack Trace:[hide]

com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Spring Application context has not been set

[Michael Cromer]

I started my deployment over after my response, and this time, when I got a 500 error, I went to the site in a new browser session, and I was able to complete the deployment. Not sure why that was the difference-maker, but here we are. My ultimate goal is to upgrade Confluence to 7.18.x and Postgres 14.x, so I think I'm halfway there.

Thank you.

Mike.

[Michael Cromer]

I've done several successful deployments now with multiple Helm chart versions of Confluence and Postgresql, and my challenge was actually not waiting long enough for Postgres to fully initialize and be ready before launching my Confluence Helm chart. New browser had nothing to do with it.