Confluence had problems starting up

Question

Hi together, We have Confluence running in a Cluster. On the first node the Confluence runs without error after installation but on the second node we get this message when I try the url direct from the server Has anyone any ideas what we can check: 2023-02-16 08:20:12,469 ERROR [Catalina-utility-1] [atlassian.config.db.DatabaseHelper] closeConnectionOrLog Connection was null. We could not successfully connect to the specified database 2023-02-16 08:20:12,469 ERROR [Catalina-utility-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:dblowercheck (Check to ensure the database correctly supports lower() for non-ASCII characters) org.hibernate.HibernateException: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: query_wait_timeout at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:63) at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.configure(DelegatingHikariConnectionProvider.java:53) at net.sf.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:84) at com.atlassian.config.db.DatabaseHelper.getConnection(DatabaseHelper.java:111) at com.atlassian.config.db.DatabaseHelper.setDatabaseLowerProperty(DatabaseHelper.java:45) at com.atlassian.config.db.DatabaseLowerPropertyCheck.startup(DatabaseLowerPropertyCheck.java:31) at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:52) at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:15) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4766) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: query_wait_timeout at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:596) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:582) at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115) at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:59) ... 18 more Caused by: org.postgresql.util.PSQLException: FATAL: query_wait_timeout at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:316) at org.postgresql.core.SetupQueryRunner.run(SetupQueryRunner.java:55) at org.postgresql.core.v3.ConnectionFactoryImpl.runInitialQueries(ConnectionFactoryImpl.java:884) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:304) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.(PgConnection.java:247) at org.postgresql.Driver.makeConnection(Driver.java:434) at org.postgresql.Driver.connect(Driver.java:291) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561) ... 21 more Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:383) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:316) at org.postgresql.core.SetupQueryRunner.run(SetupQueryRunner.java:55) at org.postgresql.core.v3.ConnectionFactoryImpl.runInitialQueries(ConnectionFactoryImpl.java:884) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:304) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.(PgConnection.java:247) at org.postgresql.Driver.makeConnection(Driver.java:434) at org.postgresql.Driver.connect(Driver.java:291) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561) at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115) at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:59) at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.configure(DelegatingHikariConnectionProvider.java:53) at net.sf.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:84) at com.atlassian.config.db.DatabaseHelper.getConnection(DatabaseHelper.java:111) at com.atlassian.config.db.DatabaseHelper.setDatabaseLowerProperty(DatabaseHelper.java:45) at com.atlassian.config.db.DatabaseLowerPropertyCheck.startup(DatabaseLowerPropertyCheck.java:31) at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:52) at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:15) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4766) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at org.postgresql.core.PGStream.receiveChar(PGStream.java:455) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ... 34 more 2023-02-16 08:20:12,489 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service 2023-02-16 08:20:12,919 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

Nic Brough -Adaptavist- · Answer

The error you are getting is in the conversation happening over the network, which should be: * Confluence: Hi, Database. Can you do this for me? Here are my credentials; x/y/z * Database: Hi, yes, here's the result (assuming credentials ok) * Confluence: What's actually happening is: * Confluence: Hi, Database. Can you do this for me? Here are my credentials; x/y/z * Database: * Confluence: Database isn't talking to me, so I can't do anything else There are three possible causes for this Confluence has exhausted the pool of possible connections and cannot create a new one. You can check this by looking at how far through the log after startup the error occurs. Was it pretty much the first thing Confluence was doing with the database or did it get a long way through startup before failing? The database is not responding to the connection. Check the database config (including the security - sometimes databases time out when incorrect security credentials are supplied or they're configured not to talk to the requesting system) The database is not reachable - that's the networking error to check. The simplest one to check here is try to connect to the database from the database's command-line tool (temporarily) installed on the Confluence server.

DG_lkynet · Answer

Hi Bhanu,
Thank you for your kindly reply.

The first one, I had checked was the network and port security.
And I would say, this is not the cause, the settings for network are still the same, as from the other node.

Telnet and netstat looks well.

We used for the 2nd node 16 gb and 4 cpus. This is more than the minimal requirements.

Do you have another idea?

Kind regards
Dennis

Bhanu KN_ww3e6e · Answer

Hi @DG ,

Hope you are doing well :)

If the Database is set up on a different server, can you check if it's reachable from the second node? Make sure two-way communication is established and try again?

I hope the hardware requirements are met (I faced issues with 2nd node because of insufficient heap and CPUs)

Cheers,

Bhanu