Confluence Fails to load after DB Update

Question

I am running CentOS 7, and uninstalled my old version of postgresql after backing my db up. I am now running version 11, and I did a restore of the db. When I attempt to visit confluence in my browser, this is what I am seeing. https://ibb.co/XkFFMP0 Update Refreshing the page now shows this: HTTP Status 500 – Internal Server Error Type Exception Report Message java.io.IOException: java.lang.reflect.InvocationTargetException Description The server encountered an unexpected condition that prevented it from fulfilling the request. Exception java.lang.IllegalStateException: java.io.IOException: java.lang.reflect.InvocationTargetException org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:101) org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256) org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282) org.apache.catalina.webresources.Cache.getResource(Cache.java:62) org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217) org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226) org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303) org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188) com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59) com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71) com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59) com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54) com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30) com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37) com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64) com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35) com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) Root Cause java.io.IOException: java.lang.reflect.InvocationTargetException org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:236) org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162) org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307) org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97) org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256) org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282) org.apache.catalina.webresources.Cache.getResource(Cache.java:62) org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217) org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226) org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303) org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188) com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59) com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71) com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59) com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54) com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30) com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37) com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64) com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35) com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) Root Cause java.lang.reflect.InvocationTargetException java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source) java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233) org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162) org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307) org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97) org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256) org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282) org.apache.catalina.webresources.Cache.getResource(Cache.java:62) org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217) org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226) org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303) org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188) com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59) com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71) com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59) com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54) com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30) com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37) com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64) com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35) com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) Root Cause java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files) java.base/java.io.RandomAccessFile.open0(Native Method) java.base/java.io.RandomAccessFile.open(Unknown Source) java.base/java.io.RandomAccessFile.(Unknown Source) java.base/java.io.RandomAccessFile.(Unknown Source) java.base/java.util.zip.ZipFile$Source.(Unknown Source) java.base/java.util.zip.ZipFile$Source.get(Unknown Source) java.base/java.util.zip.ZipFile$CleanableResource.(Unknown Source) java.base/java.util.zip.ZipFile$CleanableResource.get(Unknown Source) java.base/java.util.zip.ZipFile.(Unknown Source) java.base/java.util.zip.ZipFile.(Unknown Source) java.base/java.util.jar.JarFile.(Unknown Source) java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source) java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233) org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162) org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307) org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97) org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256) org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282) org.apache.catalina.webresources.Cache.getResource(Cache.java:62) org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217) org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226) org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303) org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334) org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188) com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59) com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71) com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59) com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54) com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30) com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37) com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64) com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35) com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) Note The full stack trace of the root cause is available in the server logs. Apache Tomcat/9.0.33 I did run the initdb command.

Nic Brough -Adaptavist- · Answer

Ok, the core of the problem is "Jira cannot connect to the database", but you've given us enough information to track that down to why - it can't actually open the database driver.

java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)

is not 100% clear - what it says is "file not found", but what it means is "I can't open a file", not that the file is not there.

The "too many open files" is the culprit, that means the OS is limited to only opening a certain number of files (by the current user) and you have run into that limit.  It is a bit archaic, one of those things that really should just go away as it is mostly pointless as hardware and software have evolved way past it (a bit like Windows for example)

Anyway, the history of it is meh.  There's a fix written up at https://confluence.atlassian.com/confkb/too-many-open-files-error-659784628.html - it boils down to "change the ulimit for the user and restart the application"