Confluence had failed to start after upgrade from 6.4 to 6.5.2

hello,

I get this message:

"

Confluence had failed to start, but we're back up and running again

Take me to my Confluence home.

Finishing up checks

"

after upgrade from 6.4 to 6.5.2

 

What can I do?

 

5 answers

0 vote
Ann Worley Atlassian Team Dec 11, 2017

The message sounds encouraging, "we're back up and running again". what happens when you click on "Take me to my Confluence home."?

If the upgrade failed there should be an error message to that effect in the <Confluence_Home>/logs/atlassian-confluence.log.

I look forward to hearing more about the upgrade - is it a Production or test instance? Did you use an installer or archive file to upgrade?

thx Ann,
when I click on it, I move to the same error page.
I used the atlassian-confluence-6.5.2-x64.exe installer.

Because it's a production site I was little bit scared so I resumed a full image of the entire virtual machine.
Now it's running the previus 6.4 version.

I would like to upgrade again, but being aware that I suppose I will have the same issue.

Ann Worley Atlassian Team Dec 12, 2017

I recommend trying the upgrade on a test instance first; here is a doc on migrating to another server so you can use it as a test instance: Migrating Confluence Between Servers

Please review the upgrade guide before the next time so you will have the latest procedure at that time: Upgrading Confluence 

You may be interested in Confluence 6.6: Confluence 6.6 Upgrade Notes so I thought I would mention that it has been released.

hello,

 

here again. 

After I took a VM snap , I tried to upgrade the latest version 6.6.1 but the same, I get this upgrade error: (more detailed than the past 6.5.x update):

 

Confluence had problems starting up
This page is for Confluence administrators. If you're seeing this page, your Confluence administrator is probably working to restore the service.

Description Time Level Exception
Upgrade failed. Please consult the system logs for details. You will need to fix these problems, restore your database and confluence home directory to the pre upgrade state. Then retry the upgrade. Upgrade error message: Cannot update schema
2018-01-10 16:14:08 error
Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
2018-01-10 16:15:49 fatal Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection

Ann Worley Atlassian Team Jan 10, 2018

Sometimes the database related error messages in the logs are really detailed and helpful.

Please check your <confluence_home>/logs/atlassian-confluence.log logs for errors like "Cannot update schema". If you find a long ERROR stack trace, search for a "caused by:" line and please let us know what you find.

ok, I reproduce the error to another test environment, and this is the last ERROR entry:

 2018-01-16 16:02:23,684 ERROR [localhost-startStop-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.springframework.orm.hibernate.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:532)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doRollback(ConfluenceHibernateTransactionManager.java:81)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:853)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:830)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy53.rollback(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:503)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:285)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy159.initialiseTriggers(Unknown Source)
at com.atlassian.confluence.schedule.listeners.ManagedScheduledJobsLifecycle.startup(ManagedScheduledJobsLifecycle.java:24)
at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:49)
at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:17)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4852)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5313)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:119)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:239)
at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:100)
at com.atlassian.hibernate.adapter.adapters.TransactionV2Adapter.rollback(TransactionV2Adapter.java:41)
at org.springframework.orm.hibernate.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:529)
... 28 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:229)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:992)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:2908)
at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:1033)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:114)
... 32 more
Ann Worley Atlassian Team Jan 16, 2018

I think you may have found our culprit:

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.

Please see if there is a validation query set for the database as described in: Surviving Database Connection Closures

The validation query is there when you do a new install of Confluence 6.5 or above but if you upgrade from pre-6.5 (like Confluence 6.4) you can be vulnerable to the issue described in the link above.

It seems like we are making progress, I hope the validation query enables the upgrade to complete.

thx @Ann Worley, I did the first point: "Enable validation query with a direct JDBC connection" but I got the same error

 

Upgrade failed. Please consult the system logs for details. You will need to fix these problems, restore your database and confluence home directory to the pre upgrade state. Then retry the upgrade. Upgrade error message: Cannot update schema
 Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)

in the log, last ERROR entry:

 

2018-01-17 09:33:41,552 ERROR [localhost-startStop-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)
org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.springframework.orm.hibernate.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:532)

....

Caused by: net.sf.hibernate.TransactionException: Unable to rollback against JDBC Connection

Ann Worley Atlassian Team Jan 17, 2018

I have seen that error "Unable to rollback against JDBC Connection" on startup related to Oracle database drivers (Unable to start up Confluence. Fatal error during startup sequence) but you are using MS SQL. 

The MS SQL driver was changed in Confluence 6.6:  SQL Server driver update

To help isolate the issue, please check which driver is specified for your instance in the file: <Confluence_home>/confluence.cfg.xml. Please look for: 

<property name="hibernate.connection.driver_class"

I look forward to hearing whether the upgrade successfully changed your MS SQL driver or if it is running with an outdated jtds driver, perhaps.

hello @Ann Worley, I found this bad error (in both environment: production 6.4 and demo 6.6.x)

 

SQL Server detected a logical consistency-based I/O error: incorrect pageid (expected 1:22500; actual 0:0). It occurred during a read of page (1:22500) in database ID 5 at offset 0x0000000afc8000 in file 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\Confluence.mdf'. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.
Ann Worley Atlassian Team Jan 24, 2018

Hi Fabio,

The file path in the error (C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA) is a default install path for SQL Server 2008. The supported platforms for both Confluence 6.4 and 6.6.x include SQL Server 2012 and 2014 but not 2008. Please verify you are using a supported version of MS SQL Server, as version differences could cause database consistency issues.

Thanks,

Ann

I've upgraded the SQL Server from 2008R2 to 2014 without any issues.

Now I'm using SQL Server 2014. but the same errors remains.

 

from Event Viewer: Event ID 824

 

SQL Server detected a logical consistency-based I/O error: incorrect pageid (expected 1:22500; actual 0:0). It occurred during a read of page (1:22500) in database ID 5 at offset 0x0000000afc8000 in file 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\Confluence.mdf'.  Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

 

I could try to export the entire Confluence configuration, shutdown this Confluence database and then create a new one where to restore it.

 

Sounds good?

Ann Worley Atlassian Team Feb 01, 2018

It sounds good if we are on the same page:

  1. Create a new database according to Database Setup for SQL Server
  2. Install Confluence using that database (no data)
  3. Restore an XML site backup from the instance that is having the consistency issues.

So you know what to expect from the site restore, please review: Restoring a Site

If you have a different plan, please let me know, my mind is open. :)

I am not a SQL server expert so maybe you can enlighten me - I am curious what you would expect to happen if you ran the database consistency check indicated in the error message.

In some cases, for example, if unique or foreign key constraints are violated by data inconsistency, the XML backup won't restore. Please update this thread to let me know if you run into such issues.

Suggest an answer

Log in or Sign up to answer
Atlassian Community Anniversary

Happy Anniversary, Atlassian Community!

This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.

Read more
Community showcase
Sarah Goff-Dupont
Published 6 hours ago in Confluence

Think you know shares vs. @mentions in Confluence? Take this collab quiz.

To anyone who doubts that Atlassians are a little too obsessed with collaboration, and tools related thereto, let me describe a recent discussion we had (which took place on our internal Confluence, ...

71 views 2 3
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you