Confluence Fails to load after DB Update

Andy June 19, 2021

I am running CentOS 7, and uninstalled my old version of postgresql after backing my db up.  I am now running version 11, and I did a restore of the db.  When I attempt to visit confluence in my browser, this is what I am seeing.  

https://ibb.co/XkFFMP0

Update

Refreshing the page now shows this:

HTTP Status 500 – Internal Server Error


Type Exception Report

Message java.io.IOException: java.lang.reflect.InvocationTargetException

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

java.lang.IllegalStateException: java.io.IOException: java.lang.reflect.InvocationTargetException
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:101)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Root Cause

java.io.IOException: java.lang.reflect.InvocationTargetException
	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:236)
	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Root Cause

java.lang.reflect.InvocationTargetException
	java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
	java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233)
	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Root Cause

java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)
	java.base/java.io.RandomAccessFile.open0(Native Method)
	java.base/java.io.RandomAccessFile.open(Unknown Source)
	java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile$Source.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile$Source.get(Unknown Source)
	java.base/java.util.zip.ZipFile$CleanableResource.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile$CleanableResource.get(Unknown Source)
	java.base/java.util.zip.ZipFile.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile.<init>(Unknown Source)
	java.base/java.util.jar.JarFile.<init>(Unknown Source)
	java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
	java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233)
	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Note The full stack trace of the root cause is available in the server logs.


Apache Tomcat/9.0.33

I did run the initdb command. 

1 answer

0 votes
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
June 19, 2021

Ok, the core of the problem is "Jira cannot connect to the database", but you've given us enough information to track that down to why - it can't actually open the database driver.

java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)

is not 100% clear - what it says is "file not found", but what it means is "I can't open a file", not that the file is not there. 

The "too many open files" is the culprit, that means the OS is limited to only opening a certain number of files (by the current user) and you have run into that limit.  It is a bit archaic, one of those things that really should just go away as it is mostly pointless as hardware and software have evolved way past it (a bit like Windows for example)

Anyway, the history of it is meh.  There's a fix written up at https://confluence.atlassian.com/confkb/too-many-open-files-error-659784628.html - it boils down to "change the ulimit for the user and restart the application"

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events