Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

502 Bad Gateway after Cloud Migration

Dirk Wunderlich December 18, 2021

Hello Community,

We have been experiencing the following problem since last night:

We have migrated several Spaces to the cloud. This has been done last night.
Now I wanted to look at our locally installed Confluence and got the error 502 - Bad Gateway.
I restarted the server, but without success.

What is the best analysis path I can take here? What logs might be helpful?

I have attached in the contents of our server.xml, as well as the nginx.conf.dpkg-dist here once:

server.xml:

<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">
<!--
==============================================================================================================
DEFAULT - Direct connector with no proxy, for unproxied HTTP access to Confluence.

If using a http/https proxy, comment out this connector.
==============================================================================================================
-->
<!--
<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"/>
-->
<!--
==============================================================================================================
HTTP - Proxying Confluence via Apache or Nginx over HTTP

If you're proxying traffic to Confluence over HTTP, uncomment the connector below and comment out the others.
Make sure you provide the right information for proxyName and proxyPort.

For more information see:
Apache - https://confluence.atlassian.com/x/4xQLM
nginx - https://confluence.atlassian.com/x/TgSvEg

==============================================================================================================
-->

<!--
<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="http" proxyName="<subdomain>.<domain>.com" proxyPort="80"/>
-->

<!--
==============================================================================================================
HTTPS - Direct connector with no proxy, for unproxied HTTPS access to Confluence.

For more info see https://confluence.atlassian.com/x/s3UC
==============================================================================================================
-->

<!--
<Connector port="8443" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25"
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLSv1.2" sslEnabledProtocols="TLSv1.2" SSLEnabled="true"
URIEncoding="UTF-8" keystorePass="<MY_CERTIFICATE_PASSWORD>"/>
-->

<!--
==============================================================================================================
HTTPS - Proxying Confluence via Apache or Nginx over HTTPS

If you're proxying traffic to Confluence over HTTPS, uncomment the connector below and comment out the others.
Make sure you provide the right information for proxyName and proxyPort.

For more information see:
Apache - https://confluence.atlassian.com/x/PTT3MQ
nginx - https://confluence.atlassian.com/x/cNIvMw
==============================================================================================================
-->


<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="https" secure="true" proxyName="confluence.connect2mobile.de" proxyPort="443"/>

<Engine name="Standalone" defaultHost="localhost" debug="0">
<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
<Manager pathname=""/>
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"
reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
</Host>
</Engine>
</Service>
</Server>

 

ngix.conf.dpkg-dist

user nginx;
worker_processes auto;

error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;


events {
worker_connections 1024;
}


http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;
#tcp_nopush on;

keepalive_timeout 65;

#gzip on;

include /etc/nginx/conf.d/*.conf;
}



We have not made any changes to the system aside from migrating to the cloud. What can I check novh?

Thanks a lot!

1 answer

1 vote
Vishwas
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 18, 2021

Hi @Dirk Wunderlich 

server.xml looks fine. 

You can start looking at confluence log files it will be like this in confluence home directory. 

path: <confluence-home>/logs/atlassian-confluence.log.

If you can paste the ERROR from log file we can see what is the exact issue.

Usually 502 error bad gateway ,that is your reverse proxy server is running but confluence has some issues starting.

Regards,

Vishwas

Dirk Wunderlich December 18, 2021

Hi @Vishwas 

thanks for your response. The log is currently uhm... a bit strange... Here the last lines. Some entries this night, after this a big nothing and for now two lines:

2021-12-18 04:07:56,848 WARN [Caesium-1-1] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2021-12-18 04:12:56,849 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2021-12-18 08:34:38,582 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.2.0 [build 8401 based on commit hash 912a577971fe711d4a876d9130277d03b009a0bf] - synchrony version 3.1.0-master-0f086db5
2021-12-18 08:34:46,526 INFO [Catalina-utility-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2021-12-18 08:34:55,415 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2021-12-18 13:14:47,445 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.2.0 [build 8401 based on commit hash 912a577971fe711d4a876d9130277d03b009a0bf] - synchrony version 3.1.0-master-0f086db5
2021-12-18 13:14:51,093 INFO [Catalina-utility-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2021-12-18 13:15:16,981 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.2.0 [build 8401 based on commit hash 912a577971fe711d4a876d9130277d03b009a0bf] - synchrony version 3.1.0-master-0f086db5
2021-12-18 13:15:20,745 INFO [Catalina-utility-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2021-12-18 13:15:25,730 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2021-12-18 13:15:42,473 INFO [Catalina-utility-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext initialized in 21727 ms
2021-12-18 13:15:43,904 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] lambda$earlyStartup$3 Plugin system earlyStartup begun
2021-12-18 13:15:45,155 WARN [Catalina-utility-1] [org.twdata.pkgscanner.ExportPackageListBuilder] logDuplicateWarning Package Scanner found duplicates for package 'com.atlassian.integration.jira.applinks' with different versions. Files: jira-integration-spi-6.2.0.jar and confluence-7.2.0.jar
'/opt/atlassian/confluence/confluence/WEB-INF/lib/jira-integration-spi-6.2.0.jar'
'/opt/atlassian/confluence/confluence/WEB-INF/lib/confluence-7.2.0.jar'
2021-12-18 13:15:48,358 WARN [FelixStartLevel] [extender.internal.support.ExtenderConfiguration] addDefaultDependencyFactories Gemini Blueprint extensions bundle not present, annotation processing disabled.

 

The last ERROR we recieved this night at 03:15:

2021-12-18 03:15:50,379 ERROR [http-nio-8090-exec-11] [confluence.plugins.synchrony.SynchronyContextProvider] getContextMap
-- url: /pages/createpage-entervariables.action | traceId: 486a6ea2d1d82995 | userName: anonymous | action: createpage-entervariables
2021-12-18 03:15:51,547 ERROR [http-nio-8090-exec-8] [confluence.plugins.synchrony.SynchronyContextProvider] getContextMap
-- url: /pages/createpage-entervariables.action | traceId: ec6fb2917277e678 | userName: anonymous | action: createpage-entervariable

Thank you and best regards

Dirk

Vishwas
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 18, 2021

Hi @Dirk Wunderlich 

This looks odd, the recent logs shows INFO and WARN levels. Seems no ERROR recently.

Can you check the status of the nginx services and send the results of ps -ef  | grep confluence

Try restarting nginx and confluence services.

Regards,

Vishwas

Dirk Wunderlich December 18, 2021

Hi @Vishwas , 

yes, its really strange :( 

nginx is running as far as I can see it.

nginx_service.jpg
I restarted it anyway, without success. JIRA and BitBucket are still running on the same machine, they work without errors.

The output from the ps -ef is the follwing:
grep.jpg

 

 

I also restarted the confluence-service, but without success.

Thank you and best regards

Dirk

Vishwas
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 18, 2021

Hi @Dirk Wunderlich 

Really not able to figure out what is the issue :(

Can you please contact Atlassian Support Team they can assist better by looking at full logs on this here https://support.atlassian.com/contact/#/

Also provide them with support zip. Steps to create support zip : 

Go to Settings(Cog Wheel)  > General Configuration > Troubleshooting and support tools (under Administration)

Regards,

Vishwas

Dirk Wunderlich December 18, 2021

Hi @Vishwas 

thank you! Yes, I wil raise a ticket. 
But unfortunately I am not able to raise a support zip, because I can't reach my confluence :D It is a  vicious circle...

Thank you and best regards!

Dirk

Vishwas
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 18, 2021

Hi @Dirk Wunderlich 

I forgot this my bad. Apologies.

Atlassian will ask for specific log files from few folders you can provide them once they reply to the ticket.

Regards,

Vishwas

Dirk Wunderlich December 18, 2021

Hi @Vishwas , 

no problem, really! Thanks for your assistance in this case!

Best regards 

Dirk

Dirk Wunderlich December 18, 2021

Hi @Vishwas 

i got it. It seems that there was some maleware on our server which claimed much off ressources. 

I cleaned it up and now Confluence is up again!

Thank you again for your support. With this I was able to walk in the correct direction.

Best regards

Dirk

Vishwas
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 19, 2021

Hi @Dirk Wunderlich 

Glad to hear the issue got fixed !!

Thanks,

Vishwas

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
TAGS
AUG Leaders

Atlassian Community Events