K8S - Confluence cannot write to home directory (but it can)

[Sean]

Hi,

After moving from Rancher to Kubernetes, I can't get confluence to start.  The error stated in the UI is: 

Setup: We can't write to your home directory

Your dedicated Confluence user needs read, write and execute permissions to the Confluence home directory /var/atlassian/confluence
See our documentation for more information on setting your home directory.

My test below shows that

• confluence is running as "daemon",
• I am logged in as "daemon",
• the folder /var/atlassian/confluence is 755 and owned by "daemon", and
• I (as daemon) can write a file to it ("why_cant_you_write_here").

bash-4.4$ ps -ef | grep java
 1 daemon 1:06 /usr/lib/jvm/java-1.8-openjdk/bin/java -Djava.util.logging.config.file=/opt/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dconfluence.context.path= -Datlassian.plugins.startup.options=-fg -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/opt/atlassian/confluence/logs/gc-2018-03-18_21-00-40.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -Dignore.endorsed.dirs= -classpath /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/confluence -Dcatalina.home=/opt/atlassian/confluence -Djava.io.tmpdir=/opt/atlassian/confluence/temp org.apache.catalina.startup.Bootstrap start


bash-4.4$ whoami
daemon


bash-4.4$ ls -l /var/atlassian/
total 0
drwxr-xr-x 1 daemon daemon 74 Mar 18 20:51 confluence


bash-4.4$ touch /var/atlassian/confluence/why_cant_you_write_here
bash-4.4$ ls -l /var/atlassian/confluence/
total 4
-rw-r--r-- 1 daemon daemon 3601 Mar 9 14:55 confluence.cfg.xml
drwxr-xr-x 1 daemon daemon 94 Mar 9 14:55 logs
drwxr-xr-x 1 daemon daemon 0 Mar 9 14:48 shared-home
-rw-r--r-- 1 daemon daemon 0 Mar 18 20:51 test
-rw-r--r-- 1 daemon daemon 0 Mar 18 21:10 why_cant_you_write_here
bash-4.4$

I don't see that this is a permissions issue, however the UI is still stating that it's a permissions issue and that it can't write here. I've tried various file/folder mode (including 777!), with no change in the error. 

During startup I get a possibly-related null pointer exception: 

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /usr/lib/jvm/java-1.8-openjdk/bin/java
2018-03-18 21:00:40,613 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
18-Mar-2018 21:00:45.287 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
18-Mar-2018 21:00:45.414 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
18-Mar-2018 21:00:45.492 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 4181 ms
18-Mar-2018 21:00:45.511 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
18-Mar-2018 21:00:45.511 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.50
18-Mar-2018 21:01:13.698 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@4f088b64]
18-Mar-2018 21:01:19.295 WARNING [localhost-startStop-2] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [605] milliseconds.
18-Mar-2018 21:01:23.398 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
18-Mar-2018 21:01:23.398 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
18-Mar-2018 21:01:23.488 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Sun Mar 18 21:01:23 GMT 2018]; root of context hierarchy
18-Mar-2018 21:01:24.492 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
2018-03-18 21:01:27,796 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.7.2 [build 7601 based on commit hash 2b9cef12215b66d36aac7298ee6598161e5b8cbb] - synchrony version 2.1.0-release-confluence_6.5-1a01ab2d
18-Mar-2018 21:01:37.890 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
18-Mar-2018 21:01:38.890 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]
18-Mar-2018 21:01:46.591 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Sun Mar 18 21:01:23 GMT 2018]; root of context hierarchy
18-Mar-2018 21:01:48.698 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
18-Mar-2018 21:01:49.094 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
18-Mar-2018 21:01:49.096 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
18-Mar-2018 21:01:51.197 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
18-Mar-2018 21:01:51.789 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 28389 ms
2018-03-18 21:02:19,497 ERROR [localhost-startStop-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
null
java.lang.NullPointerException
 at com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:43)
 at com.atlassian.confluence.impl.health.analytics.HealthCheckAnalyticsSenderSupplier.getProductionHealthCheckAnalyticsSender(HealthCheckAnalyticsSenderSupplier.java:28)
 at com.atlassian.confluence.impl.health.analytics.HealthCheckAnalyticsSenderSupplier.get(HealthCheckAnalyticsSenderSupplier.java:34)
 at com.atlassian.confluence.impl.health.analytics.HealthCheckAnalyticsSenderSupplier.get(HealthCheckAnalyticsSenderSupplier.java:16)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.sendHealthCheckResult(DefaultHealthCheckRunner.java:120)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.recordResult(DefaultHealthCheckRunner.java:109)
 at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
 at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
 at java.util.Iterator.forEachRemaining(Iterator.java:116)
 at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
 at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
 at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.executeChecksAndRecordResults(DefaultHealthCheckRunner.java:103)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.runHealthChecks(DefaultHealthCheckRunner.java:70)
 at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:177)
 at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:36)
 at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:130)
 at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:64)
 at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4861)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5322)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
2018-03-18 21:02:24,789 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
18-Mar-2018 21:02:25.095 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
18-Mar-2018 21:02:25.198 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 99705 ms

If anyone has a tip as to where to look next, I'd be happy. All of my searches have resulted in a basic howto of how to chmod/chown the directory. This hasn't been helpful, and I believe the error message is incorrect.

What exactly does confluence need to do to its home directory, besides write?

[Nic Brough -Adaptavist-]

What about daemon's own home directory?

[Sean]

It can write to specific directories there - "bin", "log", "temp" and "work" (this is the code install directory of confluence).

If I mess around with /var/atlassian/confluence such that it's not on a persistentvolume, but instead is just a part of the pod filesystem, then everything works. This means when the k8s scheduler re-schedules or scales my pod, then it loses all the attachments/plugins/cache, which of course won't fly.

[Nic Brough -Adaptavist-]

Ah, I see.  Yes, you need a stable file system for this.  You can't have the OS destroying your directories and expect it to work.

[Sean]

Sorry maybe that was unclear in my previous post.  The stable filesystem is not working.  The unstable filesystem for testing is working.

If I run the container without a PersistentVolume (testing setup), then confluence starts up correctly. It then asks for database connection and license info (since /var/atlassian/confluence was reset and empty on start). 

If I run the container with a PersistentVolume (production), then confluence fails to start because it can't write to /var/atlassian/confluence (this is the original question).

I of course can't run with my test setup, as every time it starts up it is factory-reset to a blank state. 

[Nic Brough -Adaptavist-]

Ok, yes, I misunderstood.  It is the persistent file system that is letting you down.   But that takes me back to my first question - what about daemon's home?  Can it read/write there as well as /var?

[Sean]

The home directory of the "daemon" user is /sbin, and cannot be written to by daemon. 

This is true in both the working test setup, and the non-working 'prod' setup.

Agreed it's something with the persistent volume, however as my initial report shows, the daemon user has full read/write here, so I'm not sure why the error is 'cannot write here'.  I can make many adjustments to the persistentvolume/persistentvolumeclaim if it turns out that's where I need to make an adjustment, but at the moment it's as open/permissive as it can be and it refuses to start with the no permission error.

[Nic Brough -Adaptavist-]

Ok, then I'm a bit stuck.  Your instance is breaking it at an OS level, Java does not do anything clever with your file systems, it's a simple read/write.