After years of functioning properly our instance became unstable this week. We run Jira and Confluence on one server and use a Apache proxy.
Jira is still working fine, but Confluence stops after about 3 minutes of running. From then on the port (8443) stops listening and the proxy returns a 503 error. The Confluence service also keeps running
There is no apparent error in the cataline.out logging and besides the obvous error (503) the apache logging also gives no clue about what is happening.
Server.xml
<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">
<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="148" minSpareThreads="10"
enableLookups="false" acceptCount="20" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
proxyName="***.nl" proxyPort="443" scheme="https"/>
<Engine name="Standalone" defaultHost="localhost" debug="0">
<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<Manager pathname="" />
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>
<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>
</Host>
</Engine>
</Service>
</Server>
Apache vhost
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerName ***.nl
# JIRA Proxy Configuration:
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
SSLProxyEngine On
ProxyRequests Off
ProxyPreserveHost Off
ProxyPass / http://localhost:8090/ retry=0 connectiontimeout=600 timeout=600 nocanon
ProxyPassReverse / http://localhost:8090/
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/***.nl/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/***.nl/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/***.nl/fullchain.pem
SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"
</VirtualHost>
</IfModule>
catalina.out
18-Apr-2019 21:19:55.700 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:55.805 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:55.907 INFO [http-nio-8090-exec-7] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:56.589 INFO [http-nio-8090-exec-3] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:56.896 INFO [http-nio-8090-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:56.964 INFO [http-nio-8090-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
18-Apr-2019 21:19:56.966 INFO [http-nio-8090-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorGrammarsSupport
18-Apr-2019 21:19:56.966 INFO [http-nio-8090-exec-10] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
18-Apr-2019 21:19:57.227 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:57.252 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:58.511 INFO [http-nio-8090-exec-2] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
18-Apr-2019 21:19:58.914 INFO [http-nio-8090-exec-11] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.8-atlassian-16 03/23/2015 10:20 PM'
apache error
[Thu Apr 18 21:20:04.086100 2019] [proxy:error] [pid 778:tid 139752279807744] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8090 (localhost) failed
[Thu Apr 18 21:20:04.086153 2019] [proxy:error] [pid 778:tid 139752279807744] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[Thu Apr 18 21:20:04.086165 2019] [proxy_http:error] [pid 778:tid 139752279807744] [client ****:15087] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://***
apache
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:02 +0200] "POST /synchrony-proxy/v1/data/Synchrony-afecb7c1-86dd-361c-bf94-86f370acf8a8/confluence-3342744?state-at=@head&state-format=type-tagged&rewrite-request=true&cached=true HTTP/1.1" 200 31 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:02 +0200] "GET ***?src=contextnavpagetreemode HTTP/1.1" 200 31 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:03 +0200] "POST /synchrony-proxy/v1/data/Synchrony-afecb7c1-86dd-361c-bf94-86f370acf8a8/confluence-3342744?state-at=@head&state-format=type-tagged&rewrite-request=true&cached=true HTTP/1.1" 503 632 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
orga.duh-events.nl:443 *** - - [18/Apr/2019:21:20:04 +0200] "GET ***?src=contextnavpagetreemode HTTP/1.1" 503 784 "https://***" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.48 Safari/537.36 Edg/74.1.96.24"
Normaly I would suspect the database, but it is running just fine (as is Jira).
Does anybody have a clue about what the problem might be? Or where to look for more logging to find the source of the problem?
