Confluence throwing exception after XFS filesystem fills up.

[Tom K.]

(Edited for Clarify)

Hey Guy's,  

After my XFS FS filled up, confluence kept crashing and eventually confluence throws this message.  Printout is:

"Confluence failed to establish a connection to your database."

I've tried the community pages but was not able to see any concrete solution to this issue.

I've since added space and restarted confluence but would like to know how to recover from this scenario.  The h2db.trace.db is quite large.  Not sure if it is in a consistent state or has been corrupted.  

Also wanted to add that I've started with a trial installation but then upgraded to a 10 user contract but have not yet migrated to an external database.

[root@atlas02 database]# ls -altri
total 28752250
13784525007254626176 -rw-r-----. 1 confluence confluence 742060032 Mar 6 01:54 h2db.mv.db
9671834734385157213 drwxr-x---. 2 confluence confluence 4096 Mar 12 02:44 .
9672442205410070239 drwx------. 19 confluence root 4096 Mar 19 00:06 ..
9273335826542073986 -rw-r-----. 1 confluence confluence 28700235424 Mar 19 00:18 h2db.trace.db
[root@atlas02 database]#

How do I recover from this scenario?  To note, there was an earlier crash of this host but not sure if that may have corrupted anything.  

Cheers,
TK

19-Mar-2019 00:08:52.984 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
19-Mar-2019 00:08:53.053 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 258025 ms
19-Mar-2019 00:08:57.930 SEVERE [http-nio-8090-exec-1] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=404, location=/fourohfour.action]
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
.
.
.
Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
.
.
.

UPDATE

I've ran the atlassian installer to replace any missing files that may have been corrupted and now get this message:

So I ran the installer with the exact same version as I had and I ended up moving forward, I guess.  It appears to have restored some missing files in the install path that appeared in the original exception message.  Now I'm receiving this message:

19-Mar-2019 21:48:21.229 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-10] (id=[1970]) has been active for [61,054] milliseconds (since [3/19/19 9:47 PM]) to serve the same request for [http://atlas02.nix.mds.xyz:8090/] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at java.lang.Object.wait(Native Method)
.
.
.
.
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

19-Mar-2019 21:49:31.674 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8090-exec-10] (id=[1970]) was previously reported to be stuck but has completed. It was active for approximately [130,393] milliseconds.
19-Mar-2019 22:39:18.283 SEVERE [http-nio-8090-exec-5] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
org.apache.jasper.JasperException: An exception occurred processing [/500page.jsp] at line [120]

117:
118: if (sysInfoService != null)
119: {
120: confluenceInfo = sysInfoService.getConfluenceInfo();
121: memoryInfo = sysInfoService.getMemoryInfo();
122: dbInfo = sysInfoService.getSafeDatabaseInfo();
123: sysinfo = GeneralUtil.convertBeanToMap(sysInfoService.getSystemProperties());

Stacktrace:
at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:607)
.
.
.
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
.
.
.
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:458)
... 62 more
Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
.
.
.
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
... 77 more

19-Mar-2019 22:39:18.926 SEVERE [http-nio-8090-exec-7] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
org.apache.jasper.JasperException: An exception occurred processing [/500page.jsp] at line [120]

117:
118: if (sysInfoService != null)
119: {
120: confluenceInfo = sysInfoService.getConfluenceInfo();
121: memoryInfo = sysInfoService.getMemoryInfo();
122: dbInfo = sysInfoService.getSafeDatabaseInfo();
123: sysinfo = GeneralUtil.convertBeanToMap(sysInfoService.getSystemProperties());

Stacktrace:
at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:607)
.
.
.
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
.
.
.
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:458)
... 62 more
Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
.
.
.

Alongside this message:

2019-03-20 07:29:04,401 WARN [C3P0PooledConnectionPoolManager[identityToken->1hge0yya116dyh3nq4wx2q|27ff40ab]-HelperThread-#1] [mchange.v2.resourcepool.BasicResourcePool] log Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@6144cf24 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
2019-03-20 07:29:05,053 WARN [C3P0PooledConnectionPoolManager[identityToken->1hge0yya116dyh3nq4wx2q|73bf9ca0]-HelperThread-#0] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@6aaf62f7 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool [90030-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
...
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: File corrupted in chunk 1706886, expected page length 4..192, got 828466228 [1.4.196/6]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
...

Anyway to fix this issue that doesn't rely on the UI in either case? The UI won't load and prints:

-------------------------------
Confluence had problems starting up
This page is for Confluence administrators. If you're seeing this page, your Confluence administrator is probably working to restore the service.

Confluence encountered the following problems during startup. You need to fix any errors before you start Confluence again.
Database: Database connection failed
Confluence failed to establish a connection to your database.
This could be because:
Your database isn't running
The configuration of your confluence.cfg.xml file is incorrect (user, password, or database URL etc.)
There is a network issue between Confluence and your database (e.g. firewall, database doesn't allow remote access etc.)

There are several other solutions you can try, review our documentation and see what works for you.

Learn more
If you're unable to fix the problems and need to contact support, we can respond faster if you provide the logs of your instance.
-------------------------------

Update Mar 23 2019:

Recovery tool gives me the following:

[root@atlas02 database]# java -cp h2*.jar org.h2.tools.Recover

Exception in thread "main" java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:500)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:135)
at org.h2.mvstore.DataUtils.readString(DataUtils.java:313)
at org.h2.mvstore.type.StringDataType.read(StringDataType.java:46)
at org.h2.mvstore.MVStoreTool.dump(MVStoreTool.java:238)
at org.h2.tools.Recover.process(Recover.java:343)
at org.h2.tools.Recover.runTool(Recover.java:195)
at org.h2.tools.Recover.main(Recover.java:158)
[root@atlas02 database]#

-rw-------. 1 confluence confluence 742060032 Mar 6 01:54 h2db.mv.db
-rw-------. 1 confluence confluence 56319543725 Mar 22 21:48 h2db.trace.db

[Stephen Sifers]

Hello Tom and welcome to the Community!

Thank you for providing great detail into your issue along with outputs of the file system and log snippets. If possible, I would suggest restoring to the most recent backup of Confluence (File systems and Database). You may find the restore process at Site Backup and Restore.

If a restore is not possible, it seems your error is related to the following KB which may be able to assist: Confluence generates Confluence is vacant error on install

The resolution within that KB is as follows:

Resolution

This issue can be fixed by confirming that hostname can be resolved to an IP address.

Either

1. Edit /etc/hosts and add the hostname

2. 127.0.0.1 localhost <hostname>

3. Add the hostname to the DNS

To confirm the hostname resolves to an IP address, ping the hostname (here hostname is ubuntu).

$ ping `cat /etc/hostname`

PING ubuntu (127.0.1.1) 56(84) bytes of data.

64 bytes from somepc (127.0.1.1): icmp_seq=1 ttl=64 time=0.027 ms

64 bytes from somepc (127.0.1.1): icmp_seq=2 ttl=64 time=0.041 ms

Additionally, once you have Confluence back up and running we need to ensure you migrate off the embedded database and onto a Supported Platforms database. Further information about database migration can be located at Migrating to Another Database.

Please let us know if you’re able to progress in restoring your Confluence instance.

Regards,
Stephen Sifers