Confluence Server in Docker fails to start after accidential Upgrade

Question

Hello, I have a problem with my Confluence server in docker container. I'm using atlassian/confluence-server Yesterday I changed something within my Confluence instance running 6.4.X (I dont remember X, but sure it was 6.4.whatever release). I wanted my Instance to be accessible from Internet behind reverse proxy. So I changed base URL to the external DNS. Then I got an popup regarding Confluence unable to verify base URL because of proxy settings maybe wrong. To fix this, I stopped docker container with docker stop confluence and removed it with docker rm confluence and since this is my only install, I removed the image: docker rmi atlassian/confluence-server Then I wanted to restart the container with this line: run --restart always -v /srv/docker/confluence:/var/atlassian/application-data/confluence -e "CATALINA_OPTS=-Xms512m -Xmx768m" -e "CATALINA_CONNECTOR_PROXYNAME=wiki.digitales.land" -e "CATALINA_CONNECTOR_PROXYPORT=443" -e "CATALINA_CONNECTOR_SCHEME=https" -e "CATALINA_CONNECTOR_SECURE=true" --memory=4G --dns=8.8.8.8 --name="confluence" -d -p 8090:8090 atlassian/confluence-server (Note that String "CATALINA_OPTS=..." was in the command line before, only proxy related settings were added) After that command (and waiting for some minutes), I tried to log in again into Confluence web login but saw this error: HTTP Status 500 - java.lang.IllegalStateException: Spring Application context has not been set type Exception report messagejava.lang.IllegalStateException: Spring Application context has not been set descriptionThe server encountered an internal error that prevented it from fulfilling this request. exception com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Spring Application context has not been set com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149) com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112) com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20) com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62) com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59) com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72) com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65) com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54) com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:31) com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44) com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64) com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) root cause java.lang.IllegalStateException: Spring Application context has not been set com.atlassian.spring.container.SpringContainerContext.getComponent(SpringContainerContext.java:48) com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33) com.atlassian.confluence.util.LazyComponentReference$Accessor.get(LazyComponentReference.java:42) com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85) com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325) com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143) com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112) com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20) com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62) com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59) com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72) com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65) com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54) com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:31) com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44) com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64) com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) noteThe full stack trace of the root cause is available in the Apache Tomcat/8.0.47 logs. I noticed that I've accidentially tried to upgrade Confluence to 6.5 because of inherit pulling a new image... So I was looking for several hours to fix that. When looking into docker logs confluence this messages came up: User is currently root. Will change directory ownership to daemon:daemon, then downgrade permission to daemon executing as current user If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out --------------------------------------------------------------------------- Using Java: /usr/lib/jvm/java-1.8-openjdk/bin/java 2017-11-09 08:54:51,397 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required --------------------------------------------------------------------------- 09-Nov-2017 08:54:51.794 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property. 09-Nov-2017 08:54:51.859 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'proxyPort' to '' did not find a matching property. 09-Nov-2017 08:54:51.861 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property. 09-Nov-2017 08:54:51.870 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property. 09-Nov-2017 08:54:51.876 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property. 09-Nov-2017 08:54:51.912 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property. 09-Nov-2017 08:54:51.934 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property. 09-Nov-2017 08:54:52.211 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"] 09-Nov-2017 08:54:52.230 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 09-Nov-2017 08:54:52.233 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 535 ms 09-Nov-2017 08:54:52.241 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone 09-Nov-2017 08:54:52.241 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.47 09-Nov-2017 08:54:54.048 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@294214c8] 09-Nov-2017 08:54:54.326 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher' 09-Nov-2017 08:54:54.326 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started 09-Nov-2017 08:54:54.334 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Thu Nov 09 08:54:54 GMT 2017]; root of context hierarchy 09-Nov-2017 08:54:54.430 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig] 2017-11-09 08:54:54,440 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.5.0 [build 7502 based on commit hash b9217199c27dd617c8d51f6186067d7767c5001b] - synchrony version 2.1.0-release-confluence_6.5-8d720dbe 09-Nov-2017 08:54:54.959 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler' 09-Nov-2017 08:54:55.005 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler] 09-Nov-2017 08:54:55.243 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload) 09-Nov-2017 08:54:55.245 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo() 09-Nov-2017 08:54:55.566 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Thu Nov 09 08:54:54 GMT 2017]; root of context hierarchy 09-Nov-2017 08:54:55.623 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler] 09-Nov-2017 08:54:55.725 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647 09-Nov-2017 08:54:55.779 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1451 ms 2017-11-09 08:54:57,740 INFO [localhost-startStop-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$1 Populating setup configuration if running with Cluster mode... 2017-11-09 08:54:57,878 ERROR [localhost-startStop-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below): Unable to bootstrap application: The database driver (com.mysql.jdbc.Driver) was not found in the class path.
This Confluence installation cannot be upgraded automatically. Please see Confluence documentation:
https://confluence.atlassian.com/display/DOC/Database+Setup+For+MySQL com.atlassian.config.bootstrap.BootstrapException: Unable to bootstrap application: The database driver (com.mysql.jdbc.Driver) was not found in the class path.
This Confluence installation cannot be upgraded automatically. Please see Confluence documentation:
https://confluence.atlassian.com/display/DOC/Database+Setup+For+MySQL at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:39) at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:127) at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:60) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-11-09 08:54:57,895 WARN [localhost-startStop-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextInitialized Not starting full plugin system due to upgrade or licensing errors 2017-11-09 08:54:58,610 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve 09-Nov-2017 08:54:58.635 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"] 09-Nov-2017 08:54:58.644 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 6411 ms Note the bold lines. I double checked that the MySQL driver is in the path ./WEB-INF/lib/ as it was already before the failure. To be sure, I also downloaded a new driver and copied the jar to that folder. Error persists.... What I tried also: https://community.atlassian.com/t5/Confluence-questions/Confluence-Server-install-in-Docker-falure/qaq-p/458625 and followed this: https://community.atlassian.com/t5/Confluence-questions/Confluence-not-starting-after-instalation/qaq-p/604520 besides other things like checking permissions on persistent folder and some more. Nothing worked... Another information aside: When that happened yesterday, I had an invalid license in place. This I already renewed today and also changed it inside confluence.cfg.xml I really need some support on this. Does someone have a suggestion? What will happen, if I'd remove confluence.cfg.xml to trigger a new installation? Will all data inside persistent folder and database be erased? Is it possible to do so and bring the last backup (database and files) back in place to get working installation back? Really looking forward to your suggestions. Regards Olaf

Andy Heinzer · Answer

Hi @Brad Ash

Thanks for calling this out.  I made a documentation bug ticket about this over in https://jira.atlassian.com/browse/SCALE-16

Sorry we've had the wrong info documented here.  We'll get this fixed.

Andy

Brad Ash · Answer

3 years later and this is still a problem, Docker Hub README says to copy the .jar file into

/opt/atlassian/confluence/lib

when it really needs to be

/opt/atlassian/confluence/confluence/WEB_INF/lib

There is no mention of WEB_INF folder anywhere on this page (https://hub.docker.com/r/atlassian/confluence-server/), yet according to the 7.x docs it's still the correct location (https://confluence.atlassian.com/doc/database-setup-for-mysql-128747.html)

thesefer · Answer

Glad to hear you were able to resolve your issue!

But path is the same as in the how to according to your post.

<installation-directory>/confluence/WEB-INF/lib