K8s - Confluence cannot write to home directory - folder permissions are correct

[Sean]

Hi,

After moving from Rancher to Kubernetes, I can't get confluence to start.  The error stated in the UI is: 

SETUP: WE CAN'T WRITE TO YOUR HOME DIRECTORY

Your dedicated Confluence user needs read, write and execute permissions to the Confluence home directory /var/atlassian/confluence
See our documentation for more information on setting your home directory.

My test below shows that

• confluence is running as "daemon",
• I am logged in as "daemon",
• the folder /var/atlassian/confluence is 755 and owned by "daemon", and
• I (as daemon) can write a file to it ("why_cant_you_write_here").

bash-4.4$ ps -ef | grep java
 1 daemon 1:06 /usr/lib/jvm/java-1.8-openjdk/bin/java -Djava.util.logging.config.file=/opt/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dconfluence.context.path= -Datlassian.plugins.startup.options=-fg -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/opt/atlassian/confluence/logs/gc-2018-03-18_21-00-40.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -Dignore.endorsed.dirs= -classpath /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/confluence -Dcatalina.home=/opt/atlassian/confluence -Djava.io.tmpdir=/opt/atlassian/confluence/temp org.apache.catalina.startup.Bootstrap start


bash-4.4$ whoami
daemon


bash-4.4$ ls -l /var/atlassian/
total 0
drwxr-xr-x 1 daemon daemon 74 Mar 18 20:51 confluence


bash-4.4$ touch /var/atlassian/confluence/why_cant_you_write_here
bash-4.4$ ls -l /var/atlassian/confluence/
total 4
-rw-r--r-- 1 daemon daemon 3601 Mar 9 14:55 confluence.cfg.xml
drwxr-xr-x 1 daemon daemon 94 Mar 9 14:55 logs
drwxr-xr-x 1 daemon daemon 0 Mar 9 14:48 shared-home
-rw-r--r-- 1 daemon daemon 0 Mar 18 20:51 test
-rw-r--r-- 1 daemon daemon 0 Mar 18 21:10 why_cant_you_write_here
bash-4.4$

I don't see that this is a permissions issue, however the UI is still stating that it's a permissions issue and that it can't write here. I've tried various file/folder mode (including 777!), with no change in the error. 

During startup I get a possibly-related null pointer exception: 

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /usr/lib/jvm/java-1.8-openjdk/bin/java
2018-03-18 21:00:40,613 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
18-Mar-2018 21:00:45.287 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
18-Mar-2018 21:00:45.414 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
18-Mar-2018 21:00:45.492 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 4181 ms
18-Mar-2018 21:00:45.511 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
18-Mar-2018 21:00:45.511 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.50
18-Mar-2018 21:01:13.698 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@4f088b64]
18-Mar-2018 21:01:19.295 WARNING [localhost-startStop-2] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [605] milliseconds.
18-Mar-2018 21:01:23.398 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
18-Mar-2018 21:01:23.398 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
18-Mar-2018 21:01:23.488 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Sun Mar 18 21:01:23 GMT 2018]; root of context hierarchy
18-Mar-2018 21:01:24.492 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
2018-03-18 21:01:27,796 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.7.2 [build 7601 based on commit hash 2b9cef12215b66d36aac7298ee6598161e5b8cbb] - synchrony version 2.1.0-release-confluence_6.5-1a01ab2d
18-Mar-2018 21:01:37.890 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
18-Mar-2018 21:01:38.890 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]
18-Mar-2018 21:01:46.591 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Sun Mar 18 21:01:23 GMT 2018]; root of context hierarchy
18-Mar-2018 21:01:48.698 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
18-Mar-2018 21:01:49.094 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
18-Mar-2018 21:01:49.096 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
18-Mar-2018 21:01:51.197 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
18-Mar-2018 21:01:51.789 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 28389 ms
2018-03-18 21:02:19,497 ERROR [localhost-startStop-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
null
java.lang.NullPointerException
 at com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:43)
 at com.atlassian.confluence.impl.health.analytics.HealthCheckAnalyticsSenderSupplier.getProductionHealthCheckAnalyticsSender(HealthCheckAnalyticsSenderSupplier.java:28)
 at com.atlassian.confluence.impl.health.analytics.HealthCheckAnalyticsSenderSupplier.get(HealthCheckAnalyticsSenderSupplier.java:34)
 at com.atlassian.confluence.impl.health.analytics.HealthCheckAnalyticsSenderSupplier.get(HealthCheckAnalyticsSenderSupplier.java:16)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.sendHealthCheckResult(DefaultHealthCheckRunner.java:120)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.recordResult(DefaultHealthCheckRunner.java:109)
 at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
 at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
 at java.util.Iterator.forEachRemaining(Iterator.java:116)
 at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
 at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
 at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.executeChecksAndRecordResults(DefaultHealthCheckRunner.java:103)
 at com.atlassian.confluence.impl.health.DefaultHealthCheckRunner.runHealthChecks(DefaultHealthCheckRunner.java:70)
 at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:177)
 at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:36)
 at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:130)
 at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:64)
 at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4861)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5322)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
2018-03-18 21:02:24,789 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
18-Mar-2018 21:02:25.095 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
18-Mar-2018 21:02:25.198 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 99705 ms

If anyone has a tip as to where to look next, I'd be happy. All of my searches have resulted in a basic howto of how to chmod/chown the directory. This hasn't been helpful, and I believe the error message is incorrect.

What exactly does confluence need to do to its home directory, besides write?

[Lauretha Rura]

Hey @Sean,

It appears to me that the issue is coming from the Java version Confluence was referring to - for now Confluence only supports Oracle JRE/JDK and OpenJDK is not supported yet.

Could you please try to install Oracle JRE/JDK at your end and use that instead of OpenJDK to run Confluence?

Hope this helps :)

[Sean]

There was another thread here which seems to have been removed. In that thread I demonstrated a "working" instance if I use the same container (including the openJDK), but don't do a persistent volume.  The problem with this is without a persistent volume, every time the container starts (or is rescheduled), it comes up without attachments/database/license settings -- not usable for anything, however it does demonstrate that everything is working except the PersistentVolume.

I can't see how changing the java version would help here, though I will keep that in mind for the future.

I'm looking for why it says it can't write to /var/atlassian/confluence, when clearly it can.

I'm not sure why I can't see the other reply here, but here's a link: 

https://community.atlassian.com/t5/Confluence-questions/K8S-Confluence-cannot-write-to-home-directory-but-it-can/qaq-p/752886#M101684

[Nic Brough -Adaptavist-]

I can see that thread (I remember I got stuck on it because there is nothing in Confluence that would do this.  It has to be something with the persistent file system)

[Mark de Bont]

For it was resolved by setting the ownerships manually on the share...