Error 404 and confluence stuck on boot

[Wanda Kozak]

Hi, 

I am getting a 404 error when migrating Confluence from On-premises to AWS.

I have first started a new confluence server and generated confluence.cfg.xml , then shut down confluence and started transferring data. Have done this for one of our other confluence servers in the past. When starting the server, it throws 404 error, looked through Catalina.out and don't see anything unusual. 

When I started confluence using ./start-confluence.sh -fg it started showing a couple of unusual things: 

• Boot stops at dispatcher and it hangs for serveral minutes
  • INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1563 ms

I have checked that my DB character set is utf8_bin, checked that my confluence folders are all owned by Confluence. My ports are open and I am not too sure what else can go wrong here. Working with Confluence 6.12.2 and MySQL on the new server with  mysql  Ver 14.14 Distrib 5.6.43, for Linux (x86_64) using  EditLine wrapper and on old one on mysql  Ver 14.14 Distrib 5.7.23, for Linux (x86_64) using  EditLine wrapper. 

Any ideas on what might be happening here? Thank you for any help!

Here is the log:

11-Apr-2019 22:26:51.146 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.

11-Apr-2019 22:26:51.197 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.

11-Apr-2019 22:26:51.261 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.

11-Apr-2019 22:26:51.265 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.

11-Apr-2019 22:26:51.292 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.

11-Apr-2019 22:26:51.309 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.

11-Apr-2019 22:26:51.579 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

11-Apr-2019 22:26:51.591 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read

11-Apr-2019 22:26:51.601 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 521 ms

11-Apr-2019 22:26:51.612 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]

11-Apr-2019 22:26:51.612 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.12

11-Apr-2019 22:26:53.272 INFO [localhost-startStop-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

11-Apr-2019 22:26:53.522 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@129f7f18]

11-Apr-2019 22:26:53.884 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'

11-Apr-2019 22:26:53.885 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started

11-Apr-2019 22:26:53.892 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Thu Apr 11 22:26:53 UTC 2019]; root of context hierarchy

11-Apr-2019 22:26:53.968 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]

2019-04-11 22:26:54,157 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.12.2 [build 7901 based on commit hash 119ca25093da5dbe5f504cc7d71e386a1aab2736] - synchrony version 2.1.0-master-e100417c

11-Apr-2019 22:26:54.613 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService  'defaultSockJsTaskScheduler'

11-Apr-2019 22:26:54.650 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]

11-Apr-2019 22:26:55.124 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()

11-Apr-2019 22:26:55.135 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)

11-Apr-2019 22:26:55.182 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]

11-Apr-2019 22:26:55.246 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Thu Apr 11 22:26:53 UTC 2019]; root of context hierarchy

11-Apr-2019 22:26:55.406 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647

11-Apr-2019 22:26:55.448 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1563 ms

11-Apr-2019 22:29:21.916 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file

11-Apr-2019 22:29:21.917 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors

11-Apr-2019 22:29:25.893 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.894 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.hsqldb.jdbc.JDBCDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.894 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.894 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.895 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.895 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.895 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.mysql.fabric.jdbc.FabricMySQLDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.895 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

11-Apr-2019 22:29:25.897 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

java.lang.Object.wait(Native Method)

java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

11-Apr-2019 22:29:25.898 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

java.lang.Object.wait(Native Method)

org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)

11-Apr-2019 22:29:25.901 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

java.lang.Object.wait(Native Method)

org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)

11-Apr-2019 22:29:25.901 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AtlassianEvent::CustomizableThreadFactory-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

11-Apr-2019 22:29:25.918 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]

11-Apr-2019 22:29:26.002 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 154397 ms

[Daniel Eads]

Hi Wanda,

Just circling back to see if you were able to proceed past this a couple weeks ago. I noticed the version you were trying to install is actually vulnerable to a security hole we sent an advisory out about. Details of the advisory, mitigation steps, and recommendations are all detailed in this article. 

Given the day you asked this question, it's quite possible the install was attacked while you were working on getting it set up. I would recommend trying again with the latest patched release of Confluence.

Let me know if you have more questions and how I can help!

Cheers,
Daniel