confluence 6.15 stopped working

[pierrexor]

Hi,

for whatever reason our confluence stopped working as in gateway timeout in nginx. Curl also doesnt get anything back, telnet can connect but nothing else. Now that happens but I am stuck because I can't find the reason.

System is clear, Java runs and starts fine. It's listening:

tcp6 0 0 :::8090 :::* LISTEN 2271/java

Log

10-Jul-2019 08:46:02.298 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server] failed to set property [debug] to [0]
10-Jul-2019 08:46:02.358 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
10-Jul-2019 08:46:02.374 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]
10-Jul-2019 08:46:02.380 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]
10-Jul-2019 08:46:02.408 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
10-Jul-2019 08:46:02.431 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
10-Jul-2019 08:46:02.778 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
10-Jul-2019 08:46:02.814 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [607] milliseconds
10-Jul-2019 08:46:02.823 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
10-Jul-2019 08:46:02.824 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.19]
10-Jul-2019 08:46:04.825 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@245979d5]
10-Jul-2019 08:46:04.983 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2019-07-10 08:46:05,766 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.15.6 [build 8100 based on commit hash e533a30abc9e1cf90ba3519e12647de186f0ee76] - synchrony version 2.1.0-release-confluence_6.15-32f7299a
10-Jul-2019 08:47:29.667 WARNING [Catalina-utility-2] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [84,678] milliseconds.
10-Jul-2019 08:47:29.850 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
10-Jul-2019 08:47:29.850 INFO [Catalina-utility-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
10-Jul-2019 08:47:29.856 INFO [Catalina-utility-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Wed Jul 10 08:47:29 UTC 2019]; root of context hierarchy
10-Jul-2019 08:47:29.904 INFO [Catalina-utility-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
10-Jul-2019 08:47:30.340 INFO [Catalina-utility-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
10-Jul-2019 08:47:30.381 INFO [Catalina-utility-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]
10-Jul-2019 08:47:30.732 INFO [Catalina-utility-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
10-Jul-2019 08:47:30.734 INFO [Catalina-utility-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
10-Jul-2019 08:47:30.761 INFO [Catalina-utility-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
10-Jul-2019 08:47:30.790 INFO [Catalina-utility-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Wed Jul 10 08:47:29 UTC 2019]; root of context hierarchy
10-Jul-2019 08:47:30.893 INFO [Catalina-utility-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
10-Jul-2019 08:47:30.915 INFO [Catalina-utility-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1065 ms

We're using nginx as proxy, nginx config is identical to the suggested one on confluence. I also checked all related config files.

server {
listen *:443;
server_name wiki.xxxxx;
ssl on;
ssl_certificate /etc/ssl/xxxxt;
ssl_certificate_key /etc/ssl/xxxxx;

location / {
client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://localhost:8090/;
}
location /synchrony {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://localhost:8091/synchrony;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}

[Gonchik Tsymzhitov]

Hi! 

can you open site directly through 8090 port ?

Cheers,

Gonchik Tsymzhitov

[pierrexor]

no, that was the comment about curl etc.

times out

[Gonchik Tsymzhitov]

I have asked to understand is it related to the nginx side or tomcat side? 

Because proxy_read_timeout  is 60s by default, and telnet works well if you tried.

and logs don't show any errors. 

If I were you, i will troubleshoot first without nginx and then with nginx

[pierrexor]

edit: I figured out whats wrong via strace

Not even proper error reporting does this software has...seriously what are the guys at Atlassian doing...