Confluence server will not start after I increased partition

[Ken Ducey]

I have confluence and jira installed on the same server.  I ran low on disk spaced, and increased the partition.

After rebooting, jira worked without an issue, but confluence will not load, and gives me an error:

"Your database Storage Engine could not be determined"

The atlassian-confluence log says:

2020-12-03 12:21:45,608 ERROR [http-nio-8090-exec-7] [[Standalone].[localhost].[/].[action]] log Servlet.service() for servlet [action] threw exception
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)

How can I check if confluence and jira are using the same DB?

What could be the issue?

[Nic Brough -Adaptavist-]

That's odd, I wouldn't expect that from a "run out of space" case unless you were installing, not just running.

Start with a look at <confluence home>/confluence.cfg.xml - that has the database connection in it. 

Then check the logs in <confluence home>/logs - the error message with "storage engine" should be repeated in there, but with a lot more information

[Ken Ducey]

Thanks,

I looked at the syncrhony, confluence and diagnostics log file, and could not find any errors relating to storage engine.

I do see that confluence is using 

jdbc:mysql://localhost/confluence

and jira is using

jdbc:mysql://localhost:3306/jiradb?useUnicode=true&amp;characterEncoding=UTF8&amp;sessionVariables=default_storage_engine=InnoDB

Should they be different?  As I mentioned, jira is working fine.

[Nic Brough -Adaptavist-]

I suspect the driver assumes the default port of 3306 for mysql which would mean they are both connected to the same database server.

I think we need to see more of the log as the connection fails.

[Ken Ducey]

There are three logs, which one would have that information?

[Ken Ducey]

This is the first error in the atlassian-confluence.log file:

2020-12-03 17:21:45,420 WARN [http-nio-8090-exec-6] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
-- traceId: 0062c323e0740be4
2020-12-03 17:21:45,421 ERROR [http-nio-8090-exec-6] [ContainerBase.[Standalone].[localhost].[/]] log Unhandled exception occurred whilst decorating page
-- traceId: 0062c323e0740be4
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

[Nic Brough -Adaptavist-]

That's probably a symptom of an earlier failure, you'll want to roll further back up the log

[Ken Ducey]

I rebooted the server. Please let me know if this is enough information, or if there is a way for me to send you the entire file.

Really appreciate your help!

2020-12-03 17:43:36,669 INFO [localhost-startStop-3] [atlassian.plugin.manager.DefaultPluginManager] shutdown Preparing to shut down the plugin system
2020-12-03 17:43:36,721 INFO [localhost-startStop-3] [atlassian.plugin.manager.DefaultPluginManager] shutdown Shutting down the plugin system
2020-12-03 17:43:36,721 WARN [localhost-startStop-3] [atlassian.plugin.servlet.DefaultServletModuleManager] onPluginFrameworkShutdownEvent PluginController passed via the PluginFrameworkShutdownEvent did not match that passed via PluginFrameworkStartedEvent
2020-12-03 17:43:36,737 WARN [localhost-startStop-3] [atlassian.plugin.jmx.JmxUtil] unregister Failed to unregister, instance not found: 
2020-12-03 17:43:37,094 INFO [localhost-startStop-3] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service
2020-12-03 17:43:37,201 INFO [localhost-startStop-3] [com.atlassian.confluence.lifecycle] shutdownCacheManager Shutting down EhCache cache manager
2020-12-03 17:43:37,208 INFO [localhost-startStop-3] [com.atlassian.confluence.lifecycle] contextDestroyed Stopping Confluence
2020-12-03 17:44:27,323 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.15.2 [build 8100 based on commit hash 4410012ac87e845516b70bc69b6f7a893eabaa5a] - synchrony version 2.1.0-master-9d112c9d
2020-12-03 17:44:35,434 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2020-12-03 17:44:43,465 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2020-12-03 17:45:28,559 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 53125 ms
2020-12-03 17:45:33,498 ERROR [localhost-startStop-1] [confluence.setup.dbcheck.MySQLChecker] checkIsolationLevel MySQL isolation level could not be read. Isolation level must be 'READ-COMMITTED'. See http://confluence.atlassian.com/x/GAtmDg
2020-12-03 17:45:34,635 ERROR [localhost-startStop-1] [atlassian.confluence.setup.BootstrapApplicationStartupListener] checkConfigurationOnStartup Your database Storage Engine could not be determined
2020-12-03 17:45:34,784 WARN [localhost-startStop-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextInitialized Not starting full plugin system due to upgrade
2020-12-03 17:45:36,606 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
2020-12-03 17:46:32,834 ERROR [http-nio-8090-exec-3] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- url: /rest/capabilities | traceId: 00c92ebbeef0a650
2020-12-03 17:46:33,660 WARN [http-nio-8090-exec-3] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
-- traceId: 00878a406bd410c0
2020-12-03 17:46:33,669 ERROR [http-nio-8090-exec-3] [ContainerBase.[Standalone].[localhost].[/]] log Unhandled exception occurred whilst decorating page
-- traceId: 00878a406bd410c0
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)

[Ken Ducey]

Could it be as simple as parameters missing from the URL in confluence.cfg.xml?

<property name="hibernate.connection.url">jdbc:mysql://localhost/confluence</property>

if this was wrong, would I get that error?

[Nic Brough -Adaptavist-]

You might, for some duff values.

However, the log has a smoking gun in it - look for the error about "READ COMMITTED"

[Ken Ducey]

I changed th connection url to  <property name="hibernate.connection.url">jdbc:mysql://localhost:3306/confluence?sessionVariables=tx_isolation='READ-COMMITTED'</property>,

but still has the same error

[Nic Brough -Adaptavist-]

What versions of Confluence and the mysql driver are in use?

I've seen combinations of them that ignore the properties and hence require that you do this setting on the mysql server.

[Ken Ducey]

how can I find that?

[Ken Ducey]

I believe it is Confluence 6.15.2 based on the readme file

the MySQL driver is 

mysql-connector-java-5.1.39-bin.jar

[Nic Brough -Adaptavist-]

Ok, Atlassian recommend a slightly later driver for Confluence 6.15 - see https://confluence.atlassian.com/conf615/database-jdbc-drivers-967338417.html - but I don't think that's the problem.

I'd like to try the fix they've got in the document the error points to - not adding to the connection string, but changing the properties. 

If those two still don't fix it, then you'll need to make the changes on the mysql server side.

[Ken Ducey]

Thank you. I will try upgrading the driver.  Will Confluence know to use the new driver?

"I'd like to try the fix they've got in the document the error points to - not adding to the connection string, but changing the properties."

what document? is it this one? https://confluence.atlassian.com/conf615/database-setup-for-mysql-967338433.html That is changing the MySQL side, correct?

[Ken Ducey]

Thanks again for your help.

I went through that doc, downloaded the driver, changed the web.xml and confluence.cfg.xml files.  I did not change anything on the Tomcat side, because I don't believe I need to. Rebooted the server, and still have the exact same error.

I am able to access MySQL from the CLI using mysql -h localhost --port=3306 -u confluence -pPASSWORD 'confluence' so I am using the correct port, etc.

Anything else that could be causing this?

[Nic Brough -Adaptavist-]

Sorry, I meant the document pointed to in the error message.  

Yes, you've found the right one there - I have to admit I've never had any luck changing the connection string parameters, I've always ended up putting the setting on the mysql server and restarting it.

[Ken Ducey]

Do you have any advice on what to do from here? I am not sure where to go.

I don't have a lot of expertise with MySQL, the server settings I checked appear to match.

Looking at the error again, "Your database Storage Engine could not be determined" - do you know anything more about that?  For example, could it be it is not finding the driver or something?

Thanks,

Ken

[Nic Brough -Adaptavist-]

Have you tried making the changes in the my.cnf file that are suggested in the document?

[Ken Ducey]

Yes, below is a copy of the file

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# This is for Atlassian confluence server modified by Ken
character-set-server=utf8
max_allowed_packet=256M
collation-server=utf8_bin
default-storage-engine=INNODB
max_allowed_packet=256M
innodb_log_file_size=2GB
transaction-isolation=READ-COMMITTED
binlog_format=row

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[Nic Brough -Adaptavist-]

Ok, that looks good to me (almost identical to the one I've got on a client server at the moment).

If you've restarted MySQL to pick up the changes, then I'm stumped.  Your database is telling Confluence, via the driver, that it is set up in a way Confluence can't use.  I don't know where to look next, other than raising it with Atlassian over at https://support.atlassian.com/contact 

[Ken Ducey]

I have a license, but it is a low level and I can't contact them directly.  I posted it in Stackoverflow.

Thanks again for your help, I do appreciate it!

Ken

[Ken Ducey]

If I deleted the confluence.cfg.xml, and rebooted the server, my understanding is it will run the installation procedure.  Could I do that and tell it to connect to the existing DB?

Also, after I rewrote the XML, file, it appears as though I am receiving a new error - in case this helps.

kbUrl : https://confluence.atlassian.com/confkb/startup-check-providing-logs-for-support-936509900.html?utm_source=Install&utm_medium=in-product&utm_campaign=csseng_fy18_q3_server_confluence_errorstate
caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'healthCheckAnalyticsSender' available
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)

[Nic Brough -Adaptavist-]

If it re-runs setup, you'll find it fails when it spots the existing database is not empty.

You could connect it to a new empty database, and then edit dbconfig.xml to swap back to the old one, but as the connection settings are being reported as wrong, I'm not sure that would help.

[Ken Ducey]

Is there any way to have confluence support take a look at this?  I looked at upgrading the license but it is a substantial increase.

[Ken Ducey]

Solved it! it was likely an issue with the /var/lib/mysql/performance_schema/ not being owned by mysql user, or that the mysql db needed to be upgraded.

1) renamed the confluence.cfg.xml file

2) when starting up, ran setup automatically, put in settings for the existing DB

3) tried to "test connection" received an error about writing to './performance_schema'  - realized MySQL was not the owner of the folder /var/lib/mysql/performance_schema/ - so I changed that and also changed the permissions

4) then received an error about 'performance_schema.session_variables' doesn't exist' an upgraded MySQL using mysql_upgrade

5) test connection was successful! hit next, and then it warned me there was an existing DB it was about to overwrite. Stopped, and went back and then just rebooted server

6) I copied the old confluence.cfg.xml file back

7) rebooted server, and it worked!

[Nic Brough -Adaptavist-]

Oh, that's fantastic!  I'm really glad you carried on digging and got to a solution, that was not an easy or obvious one to find.

I hope it's ok if I add this to my debugging notes?  I'd never thought to check permissions on the database service (I think because I almost always work with databases that are remote from the Atlassian system server and a little bit "black box" - they're either running fine or they are unreachable remotely)

[Ken Ducey]

Absolutely, that is why I posted it - the key to debugging was the "test connection" button on the setup - it gave me more specific errors that led me to the problem.