Jira core 7.7.1 dashboard slowness when behind proxy and using ssl

Brian Abston February 17, 2018

Here is my setup. We are running in AWS.. I am behind a haproxy set that looks for the url with port 9443 and then passes that to the jira server. I have Jira 7.7.1 installed on a Amazon Linux AMI. It is running nginx that is handling the SSL. All this works fine. It then passes to the connector and jira does pull up. The logon is fine... and there are no SSL errors.

 

The issue I am seeing is after the main logon the default dashboard is non-responsive. It also fails a health check for the Gadget URL feed. Everything else work fine and I am able to see all the other boards and issues without any problems. I have been working with support trying to get this resolved and have hit a road block. They said all looks correct in the nginx setup and also the server.xml file. 

 

Anyone have any suggestions on what else I can do or try to fix this?

2 comments

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 17, 2018

Dear @Brian Abston,

have you ensured, that localhost (or the one external interface of the VM) can connect to the external address, handled by haproxy?

There could be a firewall blocking this or a the proxy doesn't accept requests. If this is the case, Jira cannot communicate with itself via REST API. Then you get crazy when waiting for the dashboard.

So long

Thomas

Brian Abston February 18, 2018

So the external access that we get it to is https://jira.domain-name.com:9443

 

On the sever it self it can not curl that address. it can curl http://localhost:8080 which is the address that nginx is passing to the connector in jira.. See that detail below.

 

So nginx passes off to the connector as http://localhost:8080...

 

server {

    listen 443 ssl proxy_protocol;

    server_name jira.domainname.com;

location / {

        proxy_set_header X-Forwarded-Host $host;

        proxy_set_header X-Forwarded-Server $host;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://localhost:8080;

 

Then the server.xml looks like this:

 

<Connector port="8080"

                   maxThreads="150"

                   minSpareThreads="25"

                   connectionTimeout="20000" 

                   enableLookups="false"

                   maxHttpHeaderSize="8192"

                   protocol="HTTP/1.1"

                   useBodyEncodingForURI="true"

                   redirectPort="8443"

                   acceptCount="100"

                   disableUploadTimeout="true"

                   proxyName="jira.domainname.com:9443"

                   proxyPort="443"

                   scheme="https"

                   secure="true"

                   bindOnInit="false"/>

 

Do you have a suggestion on how to get the server to be able to resolve the external address? I will be glad to try that.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Dear @Brian Abston,

that's exactly the problem. Jira connects to the configured Base URL, and this will be in your case https://jira.domain-name.com:9443 and not http://localhost:8080.

But if you cannot connect from localhost to this url, Jira's own REST calls will fail.

Ensure, that you can curl https://jira.domain-name.com:9443 - then you problems are gone. Whats currently the error message when trying this?

So long

Thomas

Brian Abston February 19, 2018

curl: (7) Failed to connect to jira.domain-nme.com port 9443: Connection timed out

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Dear @Brian Abston,

this looks like the route to the address is blocked.

What does:

iptables -L -n -v

return?

Has nginx a setting to block request from local address?

What dose

nslookup jira.domain-name.com

return?

So long

Thomas

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Also:

nmap localhost

Which ports are open?

Brian Abston February 19, 2018

$ iptables -L -n -v
modprobe: ERROR: could not insert 'ip_tables': Operation not permitted
iptables v1.4.18: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

 

$ nmap localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2018-02-19 10:38 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00035s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
80/tcp open http
443/tcp open https
8080/tcp open http-proxy
8082/tcp open blackice-alerts

 

nslookup jira.domain-name.com
Server: 10.5.0.2
Address: 10.5.0.2#53

Non-authoritative answer:
Name: jira.domain-name.com
Address: 52.2.x.x

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Can you do this:

nmap -p9000-10000 localhost
Brian Abston February 19, 2018

Starting Nmap 6.40 ( http://nmap.org ) at 2018-02-19 10:51 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00028s latency).
All 1001 scanned ports on localhost (127.0.0.1) are closed

Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Ah!! This is wrong:

proxyName="jira.domainname.com:9443"

Your nginx is listening on 443 not 9443.

Correct (requires restart of Jira):

proxyName="jira.domainname.com"

And Base Url, configured in Jira UI

https://jira.domainname.com

 

Brian Abston February 19, 2018

So change the jira/conf/server.xml or the nginx.conf?

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Note: remember, that if the SSL cert that nginx is using is self-signed. Connection will fail, too. You then have to add this certificate to the keystore.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

server.xml

Brian Abston February 19, 2018

it is a valid cert.

Brian Abston February 19, 2018

the proxyName is currently set to what you say. I changed that last night try to fix. Where do you set the base url in that file?

Brian Abston February 19, 2018

If I change the base URL in the setting page I get this:

 

JIRA's base URL is set to https://jira.domain-name.com but you are accessing JIRA from https://jira.domain-name.com:9443.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Not in that file: The base URL is set in Jira.

Brian Abston February 19, 2018

Also still get the gadget URL Feed error

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

There is something weird. According to the info you gave me, port 9443 cannot be open?

What's the URL in your browser?

Brian Abston February 19, 2018

here is my server.xml as it stands now:

 

<Connector port="8080" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false" maxHttpHeaderSize="8192" protocol="HTTP/1.1" useBodyEncodingForURI="true" redirectPort="8443" acceptCount="100" disableUploadTimeout="true"
proxyName="jira.domain-name.com" proxyPort="9443" scheme="https" secure="true"/>

 

For the settings page that I am on now it :

https://jira.domain-name.com:9443/plugins/servlet/troubleshooting/view/ 

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

So hang on - I have to pick the config of one of my servers with nginx working - this can take some minutes ...

Brian Abston February 19, 2018

ok.. I can send you my nginx.conf as well if you like it. 

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

server.xml

        <Connector port="8080"
                   maxThreads="150"
                   minSpareThreads="25"
                   connectionTimeout="20000"
                   enableLookups="false"
                   maxHttpHeaderSize="8192"
                   protocol="HTTP/1.1"
                   useBodyEncodingForURI="true"
                   redirectPort="8443"
                   acceptCount="100"
                   disableUploadTimeout="true"
                   proxyName="jira.mycompany.com"
                   proxyPort="443"
                   scheme="https" />

 conf.d/jira.conf

server {
    listen   443 ssl;
    server_name jira.mycompany.com;
    keepalive_timeout   70;
    ssl_certificate     /etc/nginx/certs/jira.pem;
    ssl_certificate_key /etc/nginx/certs/jira.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;
    large_client_header_buffers 4 32k;
gzip on;
location / {
proxy_pass http://localhost:8080;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

So long

Thomas

Brian Abston February 19, 2018

Sorry.. but where is the conf.d/jira.conf ?

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

own file in /etc/nginx/conf.d

Brian Abston February 19, 2018

 

oh.. Mine is similar to that but no there. It's in the /etc/nginx/nginx.conf

 

server {

    listen 443 ssl proxy_protocol;

    server_name jira.domain-name.com;

    ssl_certificate /etc/pki/tls/certs/STAR_domain-name.com.crt;

    ssl_certificate_key /etc/pki/tls/private/STAR_domain-name.com.key;

 

    # NGINX usually only allows 1M per request. Increase this to JIRA's maximum attachment size (10M by default)

    client_max_body_size 10M;

 

    location / {

        proxy_set_header X-Forwarded-Host $host;

        proxy_set_header X-Forwarded-Server $host;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://localhost:8080;

Brian Abston February 19, 2018

Any ideas? I am heading into the office... it will be about 30-45 mins before I can get back in.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

This normally doesn't matter.

Brian Abston February 19, 2018

ok... Any ideas?

Brian Abston February 19, 2018

$ sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Brian Abston February 19, 2018

Do you think it's an iptables issue? I tried looking this page: https://confluence.atlassian.com/jirakb/running-jira-with-firewall-on-linux-741933610.html

 

And played around with steps 1-4 but not sure the right combination since I am using nginx with 443 and then redirects to 9443

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Dear @Brian Abston,

the firewall seems  to be fine.  can you meanwhile do a 

curl https://jira.domain-name.com

from localhost ???

If not, can you do this curl form another host than localhost?

So long

Thomas

Brian Abston February 19, 2018

curl: (7) Failed to connect to jira.domain-name.com port 443: Connection refused

 

I have added the domain name to the hosts file with the local address and seems to pull the page now without timing out. The health check still fails... instantly now but the dashboard does work and shows all the needed info. 

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Dear @Brian Abston,

but the Connection refused is still there? Adding a hostname to hosts will normally resolve only "cannot resolve hostname" issues.

So long

Thomas

Brian Abston February 19, 2018

Yes.. it is still there.  To be honest with you I am open to anything at this point. I am open to blowing it all away and trying something new if you want. I just was able to get this somewhat working on this setup. I was not able to find any decent guides. The only requirement is that It has to use ssl and has to be on port 9443.

 

Do you have any other trouble shooting ideas? any good guides to successfully install it? I can post my guide that I have been able to replicate the install between multi environments but this issues follows to all of them.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

Dear @Brian Abston,

probably I didn't get you the whole time. You want to open port 9443 to public with SSL termination and redirect it to localhost:8080?

Is the haproxy the public visible entry point before nginx?

haproxy:443--->nginx:9443--->localhost:8080   ????

Can you draw the whole environment, how the users will access Jira?

so long

Thomas

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 19, 2018

If haproxy is the entry, check first, if you can reach this host from the Jira installation. Most probably, this host is blocking all the REST requests from Jira. Then you have to open the firewall of this haproxy host. As I have written above, it is essential, that the tomcat process of Jira can access its public address itself.

Brian Abston February 19, 2018

Here is the flow.. https://jira.domain-name.com:9443 from the client... haproxy see the port 9443 and sends to the single jira host. nginx see the 443 and handles the cert. It passes to the same host port 8080... localhost:8080 in the server.xml.

 

From the jira server I am able to nslookup jira.domain-name.com and return the public address of the nat gateway for the private subnet. 

 

From the jira host I am able to curl http:localhost:8080 and 8082 (jira testing port) fine. When I try to add the domain name it will not work.  Even after adding the domain name and local private ip to the local hosts file on the jira host it will not do a curl of the domain name locally.

 

The haproxy is publicly accessible but is not really functioning like a proxy.. it is taking port numbers and passing them to the private subnet with this same port numbers but some services do have multi hosts behind it. So it is not doing the normal domain-name.com redirect to backend ports. more of a pass thru with load balancing settings based on autoscaling hosts behind it. 

Brian Abston February 19, 2018

the haproxy is the entry. The jira host should not have access back to the haproxies but does have access outbound via a default route to a NAT gateway.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 20, 2018

Then try this:

 <Connector port="8080"
maxThreads="150"
minSpareThreads="25"
connectionTimeout="20000"
enableLookups="false"
maxHttpHeaderSize="8192"
protocol="HTTP/1.1"
useBodyEncodingForURI="true"
redirectPort="8443"
acceptCount="100"
disableUploadTimeout="true"
proxyName="jira.domain-name.com"
proxyPort="9443"
scheme="https" />

 and:

server {
listen 443 ssl;
server_name <some internal public name or just an IP address>;
keepalive_timeout 70;
ssl_certificate /etc/nginx/certs/jira.pem;
ssl_certificate_key /etc/nginx/certs/jira.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;
large_client_header_buffers 4 32k;
gzip on;
location / {
proxy_pass http://localhost:8080;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

Then you need to configure haproxy to redirect its packages to  <some internal public name or just an IP address>. 

But eventually, you have to refactor the header information (proxy_set_header). It is decrypted on nginx. The setting are valid for nginx as the entry point.

But again - this will all fail, if Jira cannot connect to https://jira.domain-name.com:9443.

So long

Thomas

Brian Abston February 20, 2018

The haproxy is configured to direct traffic to the local internal IP of the jira host.

OK.. I tried your changes.. Same thing.

Still timing out on the main dashboard and fails the gadget feed url heath check. I am still not able to curl the url https://jira.domain-name.com:9443 from the local host. 

So the real question is why is it not able to resolve it's domain name? I think all the nginx and tomcat configs are correct. 

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 20, 2018

I recommend to draw a complete network diagram with all the hosts in your Amazon environment. That will probably help you understand, how the packages make there way and why you cannot curl.

Thomas Deiler
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 21, 2018

Dear @Brian Abston,

When the server need to go outbound since it is in private subnet... the default route is a NAT Gateway in the public subnet that is configured to send all traffic out to the internet. So the Jira host is not send or talking back to the haproxies for anything.

When you curl the address of you public subnet, it is no matter if NAT'ed or not. In the moment Jira likes to talk to itself, I has to go the same path as any client with a browser from outside. If this works (test with curl) then everything is fine.

Is the problem, that the localhost cannot resolve the name? Or is it resloved and there is no route (got IP but cannot connect)?

Some thoughts on that haproxy-host: This machine could have several virtual interfaces. Lets say eth0 and eth1. eth0 is configured for jira.domain-name.com. eth1 is configured for 'haproxy'. The software haproxy does traffic routing on application (OSI layer 7) level. So on this host, it is not required to configure any routing tables. Now when jira-localhost tries to call jira.domain.name.com it could get the IP of eth1 (a guess) but ha proxy is not listening on this nor is there any routing enabled to forward the packages to eth0. -> timeout.

The NAT thing can work, if the eth0 interface is visible for the NAT host. Best they are both in the same network. But they could be blocked for security reasons (a guess).

Anyhow - I am absolutely sure that your problems derive of the network configuration and not of Jira or nginx config.

The more information you can gather about your environment, the better we can help. 

So long

Thomas

Timothy
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 20, 2018

So, how do I explain this..

You (internal) -- 9443 --> HAProxy -- 443 --> nginx -- 8080 --> Tomcat

This should work fine.

However, as Jira does not sit in your network (unless if you have some kind of VPC or Direct Connect to AWS), the Jira server does not know anything about your HAProxy. When Jira does is own networking check:

Tomcat -- 9443 --> ???

The above can never complete because again, Jira is not within your internal network unless you have a Direct Connect or a VPN connection.

There are two solutions to fix this of course. The first option is to change / remove the requirement of 9443. You can change the requirement of 9443 to 443 (i.e. removing the port requirement). But if this is an organizational requirement, ¯\_(ツ)_/¯

The second option requires you to do this. Make sure that the server hosting Jira knows where to point the domain name too. You may need to mess around in your Route 53 in order to happen. Maybe a CNAME or an A record. Next item is to get your nginx to also listen to 9443 for SSL. Example:

server {
  listen 9443 ssl;
  listen [::]:9443 ssl;

  server_name myexample.com www.myexample.com;
  ssl_certificate /etc/letsencrypt/live/myexample.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/myexample.com/privkey.pem;

  # Redirect the browser to our port 443 config
  return 301 $scheme://myexample.com:443$request_uri;
}

server {
  listen 443 ssl;
  listen [::]:443 ssl;

  server_name myexample.com www.myexample.com;
  ssl_certificate /etc/letsencrypt/live/myexample.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/myexample.com/privkey.pem;
  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

  ...
}

Taken from: https://serverfault.com/questions/828130/how-to-run-nginx-ssl-on-non-standard-port

I don't have a full setup but I think the above will help. nginx is now ready to receive 9443. You might also need to tinker with your VPC's ACL or your Security Group (highly doubt it if the Jira server knows that localhost is your domain name) to allow that to happen.

Brian Abston February 21, 2018

everything we have is in aws. The haproxy is in a public subnet. It sees traffic coming from certain route53 domains and is used more as a load balancer than a proxy. So in this case for Jira.. the client enters the url  https://jira.domain-name.com:9443. The haproxy sees this traffic and routes to servers in a private was subnet. The jira has is running nginx and sees the ssl proxy protocol and validates the cert and passes to tomcat aka jira on the connector 8080. All this works fine. No SSL errors slowness or anything like that. We are seeing the gadget url feed heath check error.

 

When the server need to go outbound since it is in private subnet... the default route is a NAT Gateway in the public subnet that is configured to send all traffic out to the internet. So the Jira host is not send or talking back to the haproxies for anything. 

 

I pretty much have the same setup as what you have. I made a few of those changes and didn't have any luck.

Timothy
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 21, 2018

What about this then?

User -- 9443 --> HAProxy -- 9443 --> nginx -- 8080 --> Tomcat

and

Tomcat -- 9443 --> nginx -- 9443 -->  -- 8080 --> Tomcat

In your nginx, remove the listen on 443. Offload the SSL on 9443 and reverseProxy 9443 to 8080 without using 443.

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events