Confluence Server in Docker fails to start after accidential Upgrade Edited

Hello,

 

I have a problem with my Confluence server in docker container.

I'm using atlassian/confluence-server

Yesterday I changed something within my Confluence instance running 6.4.X (I dont remember X, but sure it was 6.4.whatever release).

I wanted my Instance to be accessible from Internet behind reverse proxy. So I changed base URL to the external DNS.

Then I got an popup regarding Confluence unable to verify base URL because of proxy settings maybe wrong.

To fix this, I stopped docker container with

docker stop confluence

and removed it with

docker rm confluence

and since this is my only install, I removed the image:

docker rmi atlassian/confluence-server

Then I wanted to restart the container with this line:

run --restart always -v /srv/docker/confluence:/var/atlassian/application-data/confluence -e "CATALINA_OPTS=-Xms512m -Xmx768m" -e "CATALINA_CONNECTOR_PROXYNAME=wiki.digitales.land" -e "CATALINA_CONNECTOR_PROXYPORT=443" -e "CATALINA_CONNECTOR_SCHEME=https" -e "CATALINA_CONNECTOR_SECURE=true" --memory=4G --dns=8.8.8.8 --name="confluence" -d -p 8090:8090 atlassian/confluence-server

 (Note that String "CATALINA_OPTS=..." was in the command line before, only proxy related settings were added)

After that command (and waiting for some minutes), I tried to log in again into Confluence web login but saw this error:

HTTP Status 500 - java.lang.IllegalStateException: Spring Application context has not been set

type Exception report

messagejava.lang.IllegalStateException: Spring Application context has not been set

descriptionThe server encountered an internal error that prevented it from fulfilling this request.

exception

com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Spring Application context has not been set
	com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
	com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
	com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:31)
	com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)

root cause

java.lang.IllegalStateException: Spring Application context has not been set
	com.atlassian.spring.container.SpringContainerContext.getComponent(SpringContainerContext.java:48)
	com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33)
	com.atlassian.confluence.util.LazyComponentReference$Accessor.get(LazyComponentReference.java:42)
	com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85)
	com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)
	com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
	com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
	com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:31)
	com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)

noteThe full stack trace of the root cause is available in the Apache Tomcat/8.0.47 logs.

I noticed that I've accidentially tried to upgrade Confluence to 6.5 because of inherit pulling a new image...

So I was looking for several hours to fix that.

When looking into

docker logs confluence

this messages came up:

User is currently root. Will change directory ownership to daemon:daemon, then downgrade permission to daemon
executing as current user
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /usr/lib/jvm/java-1.8-openjdk/bin/java
2017-11-09 08:54:51,397 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
09-Nov-2017 08:54:51.794 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
09-Nov-2017 08:54:51.859 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'proxyPort' to '' did not find a matching property.
09-Nov-2017 08:54:51.861 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
09-Nov-2017 08:54:51.870 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
09-Nov-2017 08:54:51.876 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
09-Nov-2017 08:54:51.912 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
09-Nov-2017 08:54:51.934 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
09-Nov-2017 08:54:52.211 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
09-Nov-2017 08:54:52.230 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
09-Nov-2017 08:54:52.233 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 535 ms
09-Nov-2017 08:54:52.241 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
09-Nov-2017 08:54:52.241 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.47
09-Nov-2017 08:54:54.048 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@294214c8]
09-Nov-2017 08:54:54.326 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
09-Nov-2017 08:54:54.326 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
09-Nov-2017 08:54:54.334 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Thu Nov 09 08:54:54 GMT 2017]; root of context hierarchy
09-Nov-2017 08:54:54.430 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
2017-11-09 08:54:54,440 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.5.0 [build 7502 based on commit hash b9217199c27dd617c8d51f6186067d7767c5001b] - synchrony version 2.1.0-release-confluence_6.5-8d720dbe
09-Nov-2017 08:54:54.959 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService  'defaultSockJsTaskScheduler'
09-Nov-2017 08:54:55.005 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]
09-Nov-2017 08:54:55.243 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
09-Nov-2017 08:54:55.245 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
09-Nov-2017 08:54:55.566 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Thu Nov 09 08:54:54 GMT 2017]; root of context hierarchy
09-Nov-2017 08:54:55.623 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
09-Nov-2017 08:54:55.725 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
09-Nov-2017 08:54:55.779 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1451 ms
2017-11-09 08:54:57,740 INFO [localhost-startStop-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$1 Populating setup configuration if running with Cluster mode...
2017-11-09 08:54:57,878 ERROR [localhost-startStop-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
Unable to bootstrap application: The database driver (com.mysql.jdbc.Driver) was not found in the class path.<br/>This Confluence installation cannot be upgraded automatically. Please see Confluence documentation:<br/>https://confluence.atlassian.com/display/DOC/Database+Setup+For+MySQL
com.atlassian.config.bootstrap.BootstrapException: Unable to bootstrap application: The database driver (com.mysql.jdbc.Driver) was not found in the class path.<br/>This Confluence installation cannot be upgraded automatically. Please see Confluence documentation:<br/>https://confluence.atlassian.com/display/DOC/Database+Setup+For+MySQL
        at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:39)
        at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:127)
        at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:60)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2017-11-09 08:54:57,895 WARN [localhost-startStop-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextInitialized Not starting full plugin system due to upgrade or licensing errors
2017-11-09 08:54:58,610 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
09-Nov-2017 08:54:58.635 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
09-Nov-2017 08:54:58.644 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 6411 ms

Note the bold lines. I double checked that the MySQL driver is in the path ./WEB-INF/lib/ as it was already before the failure.

To be sure, I also downloaded a new driver and copied the jar to that folder. Error persists....

What I tried also:

https://community.atlassian.com/t5/Confluence-questions/Confluence-Server-install-in-Docker-falure/qaq-p/458625

and followed this:

https://community.atlassian.com/t5/Confluence-questions/Confluence-not-starting-after-instalation/qaq-p/604520

besides other things like checking permissions on persistent folder and some more. Nothing worked...

Another information aside:

When that happened yesterday, I had an invalid license in place. This I already renewed today and also changed it inside confluence.cfg.xml

 

I really need some support on this.

Does someone have a suggestion?

 

What will happen, if I'd remove confluence.cfg.xml to trigger a new installation? Will all data inside persistent folder and database be erased? Is it possible to do so and bring the last backup (database and files) back in place to get working installation back?

 

Really looking forward to your suggestions.

Regards

Olaf

 

1 answer

As far as I have seen 6.5 requires oracle jdk but the Dockerfile/Container provided uses openjdk which is not supported.

You would have to build it yourself but this is not possible as described at 

https://confluence.atlassian.com/confkb/update-the-confluence-docker-image-to-use-oracle-jdk-829062521.html

because the source doesn't exists

Step 1/20 : FROM oracle/JDK:8
invalid reference format: repository name must be lowercase

alternative try

Step 1/20 : FROM oracle/jdk:8
pull access denied for oracle/jdk, repository does not exist or may require 'docker login' 

You would need to modify the Dockerfile to download and provide the mysql-driver, too.

Hello S K,

 

thanks for your response.

 

It turned out (after hours of debugging) that really only the mysql driver was the problem.

 

I followed the Howto for using MySQL driver and placed it into <INSTALL-DIR>/confluence/WEB-INF/lib/ as recommended (better: it was already there from before).

Didn't work.

 

So I decided to give a fresh "reinstall" with actual docker image atlassian/confluence-server:latest, an empty persistant directory and an empty database.

Then, in the Database Setup Dialog I chose MySQL, but were not able to continue. Confluence stated that it was missing MySQL connector.

I created the path mentioned above ("*/lib/*) and placed the .jar file into it. Checked also for the rigt permissions.

Restart... Nothing... Same problem.

 

Then I found this inside the configuration screen (browser):

Drop the .jar file in /opt/atlassian/confluence/confluence/WEB-INF/lib

This path is completely different from the one mentioned in the Confluence Documentation on web.

So I connected to the running docker container into a shell and copied the .jar connector file into the path mentioned above.

Exited and stopped the docker container.

Commited the changed image:

docker commit <container ID> local/confluence-server

Last step was to create a new container (see in my very first post) from my new image local/confluence-server

Because my image now contains the MySQL connector in the right place, my new container came up immediately and I was able to login again.

No data got lost!

 

Note that this can only be a workaround.

I Think that there's an error in the official image. Seems that the information where to find the MySQL connector in the (persistant) path is missing...

Hope my experiences will help others.

 

Regards

Olaf

Glad to hear you were able to resolve your issue!

But path is the same as in the how to according to your post.

 <installation-directory>/confluence/WEB-INF/lib

Suggest an answer

Log in or Sign up to answer
Atlassian Community Anniversary

Happy Anniversary, Atlassian Community!

This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.

Read more
Community showcase
Kesha Thillainayagam
Posted Apr 13, 2018 in Confluence

We want to hear how your non-technical teams are using Confluence!

Hi Community! Kesha (kay-sha) from the Confluence marketing team here! Can you share stories with us on how your non-technical (think Marketing, Sales, HR, legal, etc.) teams are using Confluen...

357 views 20 10
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you