Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Confluence Fails to load after DB Update Edited

Andy I'm New Here Jun 19, 2021

I am running CentOS 7, and uninstalled my old version of postgresql after backing my db up.  I am now running version 11, and I did a restore of the db.  When I attempt to visit confluence in my browser, this is what I am seeing.  

https://ibb.co/XkFFMP0

Update

Refreshing the page now shows this:

HTTP Status 500 – Internal Server Error


Type Exception Report

Message java.io.IOException: java.lang.reflect.InvocationTargetException

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

java.lang.IllegalStateException: java.io.IOException: java.lang.reflect.InvocationTargetException
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:101)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Root Cause

java.io.IOException: java.lang.reflect.InvocationTargetException
	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:236)
	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Root Cause

java.lang.reflect.InvocationTargetException
	java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
	java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233)
	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Root Cause

java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)
	java.base/java.io.RandomAccessFile.open0(Native Method)
	java.base/java.io.RandomAccessFile.open(Unknown Source)
	java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile$Source.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile$Source.get(Unknown Source)
	java.base/java.util.zip.ZipFile$CleanableResource.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile$CleanableResource.get(Unknown Source)
	java.base/java.util.zip.ZipFile.<init>(Unknown Source)
	java.base/java.util.zip.ZipFile.<init>(Unknown Source)
	java.base/java.util.jar.JarFile.<init>(Unknown Source)
	java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
	java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233)
	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)
	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)
	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)
	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)
	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)
	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

Note The full stack trace of the root cause is available in the server logs.


Apache Tomcat/9.0.33

I did run the initdb command. 

1 answer

0 votes

Ok, the core of the problem is "Jira cannot connect to the database", but you've given us enough information to track that down to why - it can't actually open the database driver.

java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)

is not 100% clear - what it says is "file not found", but what it means is "I can't open a file", not that the file is not there. 

The "too many open files" is the culprit, that means the OS is limited to only opening a certain number of files (by the current user) and you have run into that limit.  It is a bit archaic, one of those things that really should just go away as it is mostly pointless as hardware and software have evolved way past it (a bit like Windows for example)

Anyway, the history of it is meh.  There's a fix written up at https://confluence.atlassian.com/confkb/too-many-open-files-error-659784628.html - it boils down to "change the ulimit for the user and restart the application"

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Confluence

🏑 Atlympic Event: Confluence

Hello Community!  Quick disclaimer: We are running a contest on Community (The Atlympics!) from July 23rd - August 8th of 2021. If you are interested in participating in this contest (prizes! ...

469 views 18 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you