Jira throws error 500 on any type of SSL connection, including Tomcat, NGINX and Apache

[Konstantin G]

ave succesfully runt Jira Core Server 8.5.4 for a while with Nginx reverse proxy as SSL frontend. All was ok until recently upgraded to 8.13.0 and got stuck to problem: Jira starts throwing error 500 on login via SSL only. Access to plain HTTP works without any problem. Meanwhile, 8.13.1 released and I tried upgrade to 8.13.1, result is the same.

I checked causes, dignosis and resolutions from nearest similar case found (https://confluence.atlassian.com/jirakb/jira-server-throws-500-error-page-when-logging-in-or-visiting-dashboards-864653761.html), no match - there is no multiple dashboards with the same name, there is no corrupted application links, and no Base URL healthcheck fails due to SSL certificate using SAN in logs.

Also, scheme is 'https', proxyPort is '443', proxyName is 'jira.some.domain' at server.xml connector config.

Tried to setup Apache reverse proxy, with same result.

Tried to config Tomcat with SSL (https://confluence.atlassian.com/adminjiraserver071/running-jira-applications-over-ssl-or-https-802593051.html). Result is the same.

Is there any way to fix this except for reinstallation?

Error (/opt/atlassian/application-data/jira/log/atlassian-jira.log):

com.google.template.soy.tofu.SoyTofuException: In 'print' tag, expression "$dashboardTitle" evaluates to undefined.

019-05-1 11:00:00,284 http-nio-8080-exec-1 ERROR user1 111x111x1 aaaaa 1.1.1.1 / [c.a.j.web.servlet.InternalServerErrorServlet] {errorId=qqqqq-qqqqqq-qqqqqq-qqqqqq-qqqqq-qqqq, interpretedMsg=, cause=java.lang.RuntimeException: javax.servlet.ServletException: java.lang.RuntimeException: com.google.template.soy.tofu.SoyTofuException: In 'print' tag, expression "$dashboardTitle" evaluates to undefined., stacktrace=java.lang.RuntimeException: javax.servlet.ServletException: java.lang.RuntimeException: com.google.template.soy.tofu.SoyTofuException: In 'print' tag, expression "$dashboardTitle" evaluates to undefined.
at com.atlassian.web.servlet.plugin.DynamicAuthorizationServletForwarder.forward(DynamicAuthorizationServletForwarder.java:55) [?:?]
....

Error (browser https://jira.some.domain/jira then redirect to https://jira.some.domain/jira/secure/Dashboard.jspa):

The Tomcat server.xml has an incorrect configuration:
scheme should be 'https'
proxyName should be 'jira.some.domain'
proxyPort should be '443'

Troubleshoot


Sorry, we had some technical problems during your last operation.
Request assistance
Collect this information: when the problem occurred, after what operation (if you can)
Collect server logs using log's referral number (below)
Create a Support ZIP
Raise an issue for the Support team providing as much detail as possible, such as when the problem occurred, server logs, Support ZIP and the technical details (below).

echnical details
Log's referral number: 0d6a5c21-b0ae-4a8e-a45d-3f3816f77ecc

Cause
Referer URL: Unknown

java.lang.RuntimeException: javax.servlet.ServletException: java.lang.RuntimeException: com.google.template.soy.tofu.SoyTofuException: In 'print' tag, expression "$dashboardTitle" evaluates to undefined.
java.lang.RuntimeException: javax.servlet.ServletException: java.lang.RuntimeException: com.google.template.soy.tofu.SoyTofuException: In 'print' tag, expression "$dashboardTitle" evaluates to undefined.
at com.atlassian.web.servlet.plugin.DynamicAuthorizationServletForwarder.forward(DynamicAuthorizationServletForwarder.java:55) [?:?]
at com.atlassian.web.servlet.plugin.SanitizingServletForwarder.forward(SanitizingServletForwarder.java:32) [?:?]
at com.atlassian.web.servlet.plugin.RememberingServletForwarder.forward(RememberingServletForwarder.java:51) [?:?]
at com.atlassian.web.servlet.plugin.ResolvingServletForwarder.forward(ResolvingServletForwarder.java:36) [?:?]
at jsp.default_jsp._jspService(default_jsp.java:68) [classes/:?]

/opt/atlassian/jira/conf/server.xml:

<?xml version="1.0" encoding="utf-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener"/>
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"/>
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/>
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"/>

<Service name="Catalina">
<!--
==============================================================================================================
DEFAULT - Direct connector with no proxy for unproxied access to Jira.

If using a http/https proxy, comment out this connector.
==============================================================================================================
-->

<!-- Plain HTTP -->
<Connector port="8082" relaxedPathChars="[]|" relaxedQueryChars="[]|{}^&#x5c;&#x60;&quot;&lt;&gt;"
maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false"
maxHttpHeaderSize="8192" protocol="HTTP/1.1" useBodyEncodingForURI="true" 
redirectPort="443"
acceptCount="100" disableUploadTimeout="true" bindOnInit="false"/>
<!-- -->

<!-- NGINX Reverse Proxy -->
<!-- -->
<Connector port="8080" relaxedPathChars="[]|" relaxedQueryChars="[]|{}^&#x5c;&#x60;&quot;&lt;&gt;"
maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false"
maxHttpHeaderSize="16384" protocol="HTTP/1.1" useBodyEncodingForURI="true" 
acceptCount="100" disableUploadTimeout="true" bindOnInit="false" 
proxyName="jira.some.domain"
proxyPort="443" scheme="https" secure="true"/>
<!-- -->


<!--
<Connector port="8009" URIEncoding="UTF-8" enableLookups="false" protocol="AJP/1.3"/>
-->

<Engine name="Catalina" defaultHost="localhost">
<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

<Context docBase="${catalina.home}/atlassian-jira" path="/jira" reloadable="false" useHttpOnly="true">
<Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction"
factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/>
<Manager pathname=""/>
<JarScanner scanManifest="false"/>
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="120" />
</Context>

</Host>
<Valve className="org.apache.catalina.valves.AccessLogValve"
pattern="%a %{jira.request.id}r %{jira.request.username}r %t &quot;%m %U%q %H&quot; %s %b %D &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;%{jira.request.assession.id}r&quot;"/>
</Engine>
</Service>


</Server>

NGINX config file (/etc/nginx/conf.d/jira.conf):

# generated 2020-06-02, Mozilla Guideline v5.4, nginx 1.14.1, OpenSSL 1.1.1c, modern configuration
# https://ssl-config.mozilla.org/#server=nginx&version=1.14.1&config=modern&openssl=1.1.1c&guideline=5.4
server {
listen 443 ssl http2;
# listen [::]:443 ssl http2;

server_name jira.some.domain;

location /jira {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://jira.some.domain:8080/jira;
client_max_body_size 20M;

}

ssl_certificate /etc/nginx/jira.some.domain_chain.cer;
ssl_certificate_key /etc/nginx/jira.some.domain_decrypted.key;

ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m; # about 40000 sessions
ssl_session_tickets off;

# modern configuration
## ssl_protocols TLSv1.3;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

# HSTS (ngx_http_headers_module is required) (63072000 seconds)
# add_header Strict-Transport-Security "max-age=63072000" always;

# OCSP stapling
# ssl_stapling on;
# ssl_stapling_verify on;

# verify chain of trust of OCSP response using Root CA and Intermediate certs

# !Get it later!
ssl_trusted_certificate /etc/nginx/chain.cer;

# replace with the IP address of your resolver
resolver 127.0.0.1;
}

[Andrew Laden]

The fact that the logs are complaining about the wrong proxy and scheme settings in server.xml is certainly suspect.

Try setting up a separate connector specifically for use with the nginx proxy on a separate port.

Leave 8080 as the default, and run a connector on 8081 (or whatever) with the proxy settings, and then set your nginx to use that one. 

Maybe it will point out something.

[Navaneeth Ramakrishnan]

@Konstantin G @Andrew Laden  same issues we tried to upgrade Jira 8.5.4 to 8.13.2 results 503 error message. Which results rollback our changes. Do yo have any recommendations to resolve this issue. 

Note: we are apache httpd 2.4

06-Mar-2021 09:35:16.560 SEVERE [main] org.apache.catalina.core.StandardService.startInternal Failed to start connector [Connector[AJP/1.3-8009]]
org.apache.catalina.LifecycleException: Protocol handler start failed
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1102)