Confluence server upgrade 7.10.2 to 7.11.0

[Netset]

I have confluence 7.10.2 

and want to upgrade to the latest version: 7.11.0

I get this error on the UI:

Database: Your database connection pool size is not configured. The recommended minimum size is 60.

"Upgrade failed. Please consult the system logs for details. You will need to fix these problems, restore your database and confluence home directory to the pre upgrade state. Then retry the upgrade. Upgrade error message: Upgrade task com.atlassian.confluence.upgrade.upgradetask.DenormalisedSpacePermissionsUpgradeTask@5aa3e53b failed during the SCHEMA_UPGRADE phase due to: StatementCallback; uncategorized SQLException for SQL [DROP PROCEDURE IF EXISTS space_permission_procedure_for_denormalised_permissions;]; SQL state [HY000]; error code [1728]; Cannot load from mysql.proc. The table is probably corrupted; nested exception is java.sql.SQLException: Cannot load from mysql.proc. The table is probably corrupted"

Then I downgraded the 7.10.2 and restored my DB backup and it worked again, I tried to upgrade again I get the same error!

My confluence server is installed on Linux centOS 7 och DB is MySQL in docker on the same server! 

what I checked:

My pool size is 60 as recommended!

Please help me to find a solution! 

[Daniel Eads · Accepted solution]

Hi @Netset ,

I see you've created several questions here on Community over the last couple months about the upgrade from Confluence 7.10.2 to 7.11.0 / 7.11.1, and potentially migrating from a MySQL database to MariaDB. Do note that as indicated on the Confluence supported platforms page , MariaDB is explicitly not supported by Confluence 7.11 (per CONFSERVER-29060 ).

I see also that according to your reply on a different thread, it looks like you were able to proceed by removing three tables from the database.

Given the difficulty you've had over this upgrade and that you're on a 10 user Starter license, I do wonder if it might be easier and less time consuming for you to consider migration to our Cloud Free plan. For the low cost of free, you can still get 10 users on our free plan in Confluence Cloud , but not have to worry about application upgrades or the database and filesystem management. The Confluence Cloud Migration Assistant should be pre-loaded into your Confluence install if you want to sign up for a Free cloud instance and give the migration a try. You can do test migrations before making a decision to cut over for real, as the migration app leaves all your Server data intact during migration runs. Let me know if you have questions around Cloud Free or giving the migration assistant a try.

Cheers,
Daniel | Atlassian Community

[Netset]

Hi

I would like to migrate to the cloud! but I need to have time dedicated to reading and go through all conditions! which I don't have right now! That is why I needed to spend less time upgrading and have the latest version before doing that! 

I have MySQL 5.7 and tested with mysql8 in my test server! MariaDB was just a quick test so I am aware that MariaDB is not supported! 

because I have tested many times and the logs give me the problem in duplicated key when creating objects in those three tables, I dropped them but not knowing the impact of that! I need help with that, but surely thankful for suggesting the migration to the cloud which I already consider.

I have written many times in the community because it was impossible to find the contact to make a support issue as I have license as I am on a supported instance.

[Daniel Eads]

I understand - thanks for the context! Cloud migrations are supported on older versions - you can be as far back as Confluence 5.10 and not need to upgrade to the latest version before upgrading to Cloud. My recommendation would be to pause the upgrade work, and go directly to learning about the migration / testing that out when you have the time available. I hope things calm down for you and you are able to enjoy some rest soon!

[Netset]

I take your advice and started with testing the migration to the cloud!

But in the preparation mode, I needed to decide if I will delete disabled users. 

If I delete a user, will that delete the content that the user created? will that impact the content by any means?

[Daniel Eads]

If you delete a user on Server, the content they created will still be available but won't be linked to a user account anymore. Confluence Server will instead create a placeholder for them. From the documentation about deleting a Server user: 

When a user account is deleted:

• The user won't be able to log in.
• The user won't be included in your license count.
• People won't be able to see the user in the People directory, mention them, or select their name/username as a search filter.
• An anonymised alias will appear on any spaces or content they have created.
• They won't be listed in User Management admin screens.
• Their account is deleted and anonymised permanently, and can't be restored.
• Any content they created will be maintained. 

On the other hand, here's the impact if you leave those users as disabled rather than deleting them and then they're included in the migration assistant's run:

Users with disabled status in your server site will be migrated as active but without any product access. This means they will not be counted as active Confluence users for billing purposes.

[Netset]

Now I have made my very first attempts to migrate the site, all works fine, but I have two issues:

1- I have one failed space (can't read two attachments)

2- I have made my spaces public which is not good, how can I delete the cloud site and try again lets sya my cloude site is https://guidetest.atlassian.net/

[Daniel Eads]

1. Missing attachments - the easiest way to correct this is finding out the file path an attachment should be at in your home directory, and creating an empty file with the missing attachment's name in that folder. Detailed instructions for this procedure are available on this KB article .
2. In order to remove and retry a Cloud migration, the easiest thing to do if you're not already using the Cloud product in production is to remove the product (Confluence) from your Cloud account and re-add it. Detailed instructions for deleting the product from your account to reset the data are here - but note that it's slightly different depending on if that's the only Cloud product on your account or if you have another one already.

[Jing Kang]

cannot edit/create page anymore

content_procedure_for_denormalised_permissions does not exist

[Normann P_ Nielsen _Netic_]

https://confluence.atlassian.com/confkb/confluence-mysql-database-migration-causes-content_procedure_for_denormalised_permissions-does-not-exist-error-1072474724.html

[Netset]

Still have problem!

Server CentOS 7 latest kernel version

-----------------------------------------------------------------

Mysql: Ver 8.0.23 for Linux on x86_64

mysql settings: /etc/my.cnf

innodb_log_file_size=2GB
max_allowed_packet=256M
binlog_format=row
character-set-server=utf8mb4
collation-server=utf8mb4_bin

sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION

in mysql > 
| max_connections | 151 |
| mysqlx_max_connections | 100 |

----------------------------------------------------------------

Confluence Server configuration:

1- installation_dir/conf/server.xml

using jndi datasource connection

driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3306/confluence?useUnicode=true&characterEncoding=utf8&useSSL=false"
maxTotal="80"
maxIdle="20"

2-  installation_dir/bin/setenv.sh

CATALINA_OPTS="-Xms1024m -Xmx6000m -XX:+UseG1GC ${CATALINA_OPTS}"
JAVA_HOME="/opt/java11" #latest LTS version of java 11

-------------------------------------------------------------------

when upgrading from 7.10.2 to 7.11.1 I see in logs HOME_DIR/confluence-7.11.1/logs/atlassian-confluence.log

that it make recovery for all databsetables and then takes abit longer in:

[confluence.upgrade.recovery.ConfluenceProgressMonitor] begin TABLE_DATA: [scheduler_run_details]

then  get 

2021-03-18 11:09:31,512 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] end []
2021-03-18 11:09:31,513 INFO [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeManager] beforeUpgrade Finished generating pre-upgrade recovery file.
2021-03-18 11:09:34,273 INFO [Catalina-utility-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (N
AME, TYPE)
2021-03-18 11:09:34,312 WARN [Catalina-utility-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
->[com.atlassian.confluence.upgrade.ddl.HibernateDdlExecutor.executeDdl]: PROPAGATION_REQUIRES_NEW,ISOLATION_DEFAULT (Session #347931867)
2021-03-18 11:09:34,353 ERROR [Catalina-utility-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] launchUpgrades Upgrade failed, application will not start: Upgrade task com.atlassian.confluence.up
grade.upgradetask.DenormalisedSpacePermissionsUpgradeTask@3f951504 failed during the SCHEMA_UPGRADE phase due to: StatementCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on D
ENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'denormalised_sid_name_type_uniq_idx'
com.atlassian.confluence.upgrade.UpgradeException: Upgrade task com.atlassian.confluence.upgrade.upgradetask.DenormalisedSpacePermissionsUpgradeTask@3f951504 failed during the SCHEMA_UPGRADE phase due to: Statem
entCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
key name 'denormalised_sid_name_type_uniq_idx'

.

.

.

Caused by: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com
.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'denormalised_sid_name_type_uniq_idx'

.

.

.

at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376)
... 34 more
2021-03-18 11:09:34,356 ERROR [Catalina-utility-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] launchUpgrades 1 errors were encountered during upgrade:
2021-03-18 11:09:34,357 ERROR [Catalina-utility-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] launchUpgrades 1: StatementCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'denormalised_sid_name_type_uniq_idx'
2021-03-18 11:09:34,366 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] lambda$lateStartup$4 Plugin system lateStartup begun
2021-03-18 11:09:37,549 INFO [ThreadPoolAsyncTaskExecutor::Thread 41] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-03-18 11:09:38,891 WARN [ThreadPoolAsyncTaskExecutor::Thread 55] [addons.analytics.scheduler.DataRetentionScheduleManagerImpl] afterPropertiesSet Initializing Data Retention Schedule with frequency: 60000 milliseconds, batch size: 10000
2021-03-18 11:09:39,204 WARN [ThreadPoolAsyncTaskExecutor::Thread 55] [addons.analytics.scheduler.EventLimiterScheduleManagerImpl] afterPropertiesSet Initializing Event Limiter Schedule with frequency: 60000 milliseconds

then it repeats the database tables recovery and then

2021-03-18 11:10:31,086 INFO [AtlassianEvent::CustomizableThreadFactory-2] [atlassian.confluence.event.MonitorableCallerRunsPolicy] rejectedExecution Asynchronous queue is full. The task will be processed synchr
onously in the current thread instead. We'll print this message every 60 seconds until there's enough space in queue to start processing messages asynchronously again. Executor: [java.util.concurrent.ThreadPoolE
xecutor@5aab81d9[Running, pool size = 2, active threads = 2, queued tasks = 256, completed tasks = 564]]
2021-03-18 11:11:12,872 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] end TABLE_DATA: [scheduler_run_details]
2021-03-18 11:11:12,873 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] begin TABLE_DATA: [storageenginetest]
2021-03-18 11:11:12,874 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] end TABLE_DATA: [storageenginetest]
2021-03-18 11:11:12,874 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] begin TABLE_DATA: [user_mapping]
2021-03-18 11:11:12,875 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] end TABLE_DATA: [user_mapping]
2021-03-18 11:11:12,876 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] end TABLES_DATA: []
2021-03-18 11:11:12,876 INFO [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] end []
2021-03-18 11:11:12,877 INFO [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeManager] entireUpgradeFinished Finished generating post-upgrade recovery file.
2021-03-18 11:11:12,877 INFO [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] entireUpgradeFinished Upgrade completed successfully
2021-03-18 11:11:14,717 INFO [Catalina-utility-1] [confluence.admin.actions.SystemInfoOnStartup] startup
--------------------------
Build Information
--------------------------
maxUsers = 10
installationDate = Thu Feb 21 13:59:56 CET 2019

=====================================

when I look in the GUI:

Confluence had problems starting up

This page is for Confluence administrators. If you're seeing this page, your Confluence administrator is probably working to restore the service.
Database: Your database connection pool size is not configured. The recommended minimum size is 60.

Learn more
Description Time Level Exception
Upgrade failed. Please consult the system logs for details. You will need to fix these problems, restore your database and confluence home directory to the pre upgrade state. Then retry the upgrade. Upgrade error message: Upgrade task com.atlassian.confluence.upgrade.upgradetask.DenormalisedSpacePermissionsUpgradeTask@3f951504 failed during the SCHEMA_UPGRADE phase due to: StatementCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'denormalised_sid_name_type_uniq_idx'

==========================================

I can send the log via mail if it is needed!

I have tried to delete the following Tables in database:

DENORMALISED_SPACE_VIEW_PERMISSIONS
DENORMALISED_SPACE_EDIT_PERMISSIONS
DENORMALISED_SID

and then I could upgrade, but I don't really know the impact of removing these tables!

[Netset]

Solution!

I have upgraded using another server with new version of MariaDB (with my backup database restored to it ) and it works!

[Brant Schroeder]

@Netset 

If your connection pool size has been properly set then you most likely just need to increase the number of allowed connection.  Please follow the documentation here to resolve this issue.

https://confluence.atlassian.com/confkb/startup-check-database-connection-pool-size-960713815.html

[Netset]

The problem is still here in the production server!

I have followed the correct values of database! when I use the Mysql in docker the upgrade doesn't fix the problem of:

->[com.atlassian.confluence.upgrade.ddl.HibernateDdlExecutor.executeDdl]: PROPAGATION_REQUIRES_NEW,ISOLATION_DEFAULT (Session #190623470)
2021-03-04 16:03:52,276 ERROR [Catalina-utility-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] launchUpgrades Upgrade failed, application will not start: Upgrade task com.atlassian.confluence.upgrade.upgradetask.DenormalisedSpacePermissionsUpgradeTask@76e0a
831 failed during the SCHEMA_UPGRADE phase due to: StatementCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key na
me 'denormalised_sid_name_type_uniq_idx'
com.atlassian.confluence.upgrade.UpgradeException: Upgrade task com.atlassian.confluence.upgrade.upgradetask.DenormalisedSpacePermissionsUpgradeTask@76e0a831 failed during the SCHEMA_UPGRADE phase due to: StatementCallback; bad SQL grammar [create unique index denormalised
_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'denormalised_sid_name_type_uniq_idx'
at com.atlassian.confluence.upgrade.AbstractUpgradeManager.executeUpgradeStep(AbstractUpgradeManager.java:262)
at com.atlassian.confluence.upgrade.AbstractUpgradeManager.runSchemaUpgradeTasks(AbstractUpgradeManager.java:218)
at com.atlassian.confluence.upgrade.AbstractUpgradeManager.upgrade(AbstractUpgradeManager.java:164)
at com.atlassian.confluence.plugin.PluginFrameworkContextListener.launchUpgrades(PluginFrameworkContextListener.java:122)
at com.atlassian.confluence.plugin.PluginFrameworkContextListener.contextInitialized(PluginFrameworkContextListener.java:82)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5177)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [create unique index denormalised_sid_name_type_uniq_idx on DENORMALISED_SID (NAME, TYPE)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplica
te key name 'denormalised_sid_name_type_uniq_idx'
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:93)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1443)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:388)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:418)
at com.atlassian.confluence.upgrade.ddl.HibernateDdlExecutor.executeDdlStatements(HibernateDdlExecutor.java:92)
at com.atlassian.confluence.upgrade.ddl.HibernateDdlExecutor.executeDdl(HibernateDdlExecutor.java:83)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
:

[Brant Schroeder]

What version of mySQL are you using?  Here is a list of supported platforms. https://confluence.atlassian.com/doc/supported-platforms-207488198.html#SupportedPlatforms-Databases   If you are not on a supported DB you will need to update the DB instance first.

My guess is that you are on a supported instance and that this is just an error in the DB scheme, if that is the case you will have to submit a support issue to get the DB scheme structure corrected so the Atlassian upgrade scripts can run. https://support.atlassian.com/contact/

[Netset]

Should I open another case for the same question?

The link you gave to med lead to the same environment as we are now?

[Brant Schroeder]

You are currently in community and we can only do so much.  If you have a paid license then you will be able to open a support case where Atlassian techs will be able to help you.  Are you on a supported version of mySQL?

[Netset]

Yes I am a supported Mysql 

[Brant Schroeder]

@Netset is this the same issue? https://community.atlassian.com/t5/Confluence-questions/Confluence-server-upgrade-7-10-2-to-7-11-0/qaq-p/1629617

If this is the same issue there is a known issue that I just found that might be related https://jira.atlassian.com/browse/CONFSERVER-57869

[Brant Schroeder]

@Netset 

I escalated to Atlassian to see if this is a known issue.  They will also be able to tell you about the tables you are asking about deleting.

[Netset]

yes, thanks! I need to say that after deleting the tables and run the upgrade! the tables are created again!

[Normann P_ Nielsen _Netic_]

https://confluence.atlassian.com/confkb/install-upgrade-confluence-7-11+-fails-with-error-you-do-not-have-the-super-privilege-and-binary-logging-is-enabled-1044096915.html