Service Unavailable after making space accessible by anounymous users

[Gus Johansson]

Hi,

I tried the Mention function in the comment section and our Confluence Server 6.14 just stopped working. Now I can not start it again.

I have reset the service and restarted the server and still same.

________________________________________

Edit:

I have now caused Confluence to crash a second time. This is caused by the settings that make a space available for anonymous users. I have a test space, I opened it up for anonymous access and site crashed. When I try to enter the site i get: 

"Http/1.1 Service Unavailable"

Service is running. A reset of service does not work. A restart of server does not work.

All I see in the logs are:

04-Apr-2019 12:59:24.102 INFO [https-jsse-nio2-8443-exec-23] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 12:59:24.133 INFO [https-jsse-nio2-8443-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 12:59:25.430 INFO [https-jsse-nio2-8443-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 13:00:13.709 INFO [https-jsse-nio2-8443-exec-11] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

Here are logs from first incident, they dont seem to interrupt the service:

03-Apr-2019 17:15:50.911 WARNING [localhost-startStop-4] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

03-Apr-2019 17:15:50.942 WARNING [localhost-startStop-4] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AtlassianEvent::CustomizableThreadFactory-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

....

03-Apr-2019 17:15:51.005 SEVERE [localhost-startStop-4] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [com.atlassian.confluence.cache.TransactionalCacheFactory$1] (value [com.atlassian.confluence.cache.TransactionalCacheFactory$1@4b0f26a2]) and a value of type [com.atlassian.confluence.cache.TransactionalCacheFactory.DelegateCacheSynchronization] (value [com.atlassian.confluence.cache.TransactionalCacheFactory$DelegateCacheSynchronization@17b47241]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

[Gus Johansson]

I have now caused Confluence to crash a second time. This is caused by the settings that make a space available for anonymous users. I have a test space, I opened it up for anonymous access and site crashed. When I try to enter the site i get: 

"Http/1.1 Service Unavailable"

Service is running. A reset of service does not work. A restart of server does not work.

All I see in the logs are:

04-Apr-2019 12:59:24.102 INFO [https-jsse-nio2-8443-exec-23] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 12:59:24.133 INFO [https-jsse-nio2-8443-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 12:59:25.430 INFO [https-jsse-nio2-8443-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 13:00:13.709 INFO [https-jsse-nio2-8443-exec-11] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

Please assist.

[Gus Johansson]

We are now up and running again. We restored the database prior to my Mention comment. 

* Could it be the mention comment that caused this? Very good question. We do not know.

Here are logs from the working startup:

04-Apr-2019 09:52:53.800 SEVERE [https-jsse-nio2-8443-Acceptor-0] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.util.concurrent.ExecutionException: java.io.IOException: The specified network name is no longer available.

at sun.nio.ch.PendingFuture.get(PendingFuture.java:185)
at org.apache.tomcat.util.net.Nio2Endpoint.serverSocketAccept(Nio2Endpoint.java:339)
at org.apache.tomcat.util.net.Nio2Endpoint.serverSocketAccept(Nio2Endpoint.java:56)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: The specified network name is no longer available.

at sun.nio.ch.Iocp.translateErrorToIOException(Iocp.java:309)
at sun.nio.ch.Iocp.access$700(Iocp.java:46)
at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:399)
... 1 more

04-Apr-2019 09:54:34.349 INFO [https-jsse-nio2-8443-exec-12] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:53.040 INFO [https-jsse-nio2-8443-exec-13] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:53.149 INFO [https-jsse-nio2-8443-exec-12] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:53.399 INFO [https-jsse-nio2-8443-exec-12] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
04-Apr-2019 09:54:53.415 INFO [https-jsse-nio2-8443-exec-12] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
04-Apr-2019 09:54:53.415 INFO [https-jsse-nio2-8443-exec-12] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
04-Apr-2019 09:54:53.431 INFO [https-jsse-nio2-8443-exec-13] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
04-Apr-2019 09:54:53.477 INFO [https-jsse-nio2-8443-exec-13] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
04-Apr-2019 09:54:53.477 INFO [https-jsse-nio2-8443-exec-13] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
04-Apr-2019 09:54:53.977 INFO [https-jsse-nio2-8443-exec-4] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:54.196 INFO [https-jsse-nio2-8443-exec-16] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:54.602 INFO [https-jsse-nio2-8443-exec-17] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:54.806 INFO [https-jsse-nio2-8443-exec-1] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:55.618 INFO [https-jsse-nio2-8443-exec-19] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:56.119 INFO [https-jsse-nio2-8443-exec-7] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:56.243 INFO [https-jsse-nio2-8443-exec-13] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:57.212 INFO [https-jsse-nio2-8443-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:54:57.368 INFO [https-jsse-nio2-8443-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
04-Apr-2019 09:54:57.368 INFO [https-jsse-nio2-8443-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
04-Apr-2019 09:54:57.368 INFO [https-jsse-nio2-8443-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
04-Apr-2019 09:55:00.720 INFO [https-jsse-nio2-8443-exec-24] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:55:12.455 INFO [https-jsse-nio2-8443-exec-3] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:55:13.049 INFO [https-jsse-nio2-8443-exec-5] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:55:13.127 INFO [https-jsse-nio2-8443-exec-1] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:55:13.268 INFO [https-jsse-nio2-8443-exec-17] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:55:18.248 INFO [https-jsse-nio2-8443-exec-19] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:55:18.466 WARNING [https-jsse-nio2-8443-exec-19] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://confluence.xenit.se/rest/tinymce/1/content/1245203/comment?actions=true, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
04-Apr-2019 09:55:19.841 INFO [https-jsse-nio2-8443-exec-5] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:56:03.485 INFO [https-jsse-nio2-8443-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
04-Apr-2019 09:59:21.608 INFO [https-jsse-nio2-8443-exec-5] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'

[Gus Johansson]

This is what I see in my logs when attempting to start the service:

04-Apr-2019 08:45:47.449 SEVERE [https-jsse-nio2-8443-Acceptor-0] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.util.concurrent.ExecutionException: java.io.IOException: The specified network name is no longer available.

at sun.nio.ch.PendingFuture.get(PendingFuture.java:185)
at org.apache.tomcat.util.net.Nio2Endpoint.serverSocketAccept(Nio2Endpoint.java:339)
at org.apache.tomcat.util.net.Nio2Endpoint.serverSocketAccept(Nio2Endpoint.java:56)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: The specified network name is no longer available.

at sun.nio.ch.Iocp.translateErrorToIOException(Iocp.java:309)
at sun.nio.ch.Iocp.access$700(Iocp.java:46)
at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:399)
... 1 more