Good initial comms but then 403 Error Jira Cloud to Bamboo App Link

[upsampled]

Summary

I am trying to get an application link between my origination Bamboo Instance and our Atlassian JIRA-Cloud account. The application link from JIRA-Clopud to bamboo is broken and gives error 403. Without more information from JIRA-Cloud I cannot figure out the cause of the problem. Any idea how to fix this?

Details

We are running bamboo behind nginx following these steps.

The link form bamboo to JIRA-cloud looks good, but I am having trouble with the reverse. Initially it looks good, JIRA-Cloud looks like it is thinking and nginx shows good responces (nginx log below)

192.168.xxx.xxx - - [10/Jun/2019:08:49:13 -0400] "OPTIONS /bamboo/plugins/servlet/oauth/consumer-info?_=000000000000 HTTP/1.1" 200 0 "https://xxxxxxx.atlassian.net/plugins/servlet/applinks/listApplicationLinks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
192.168.xxx.xxx - - [10/Jun/2019:08:49:13 -0400] "GET /bamboo/plugins/servlet/oauth/consumer-info?_=00000000 HTTP/1.1" 200 643 "https://xxxxxxx.atlassian.net/plugins/servlet/applinks/listApplicationLinks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

Then this suddenly stop with no further information from nginx. I get the following from JIRA-Cloud:

Unexpected response

We received an unexpected response from XXXXXX Bamboo. If the remote application is behind a proxy, check that the application is operational. Troubleshoot this for other possible causes.

403: Forbidden

To see the full response content look at your application logs.

Looking up the error you get the following

HTTP 403: Forbidden This usually indicates an authentication failure. However application links report authentication failures differently from a 403, so this suggests a problem with the proxy configuration instead.

The remote application is behind a misconfigured proxy. Check the local application logs for the <response body>. If the application has been configured with an outbound proxy, the remote application URL can be added to the nonProxyHosts to allow the application link to bypass it. See How to Configure Outbound HTTP and HTTPS Proxy for your Atlassian application nonProxyHosts may be configured incorrectly and non-functional. See: Java Option http.nonProxyHosts does not work JAVA Option '-Dhttp.nonProxyHosts' Does Not Work on Windows OS

The application logs just show what the nginx logs show, good HTTP requests. There is one error but it looks like an intentional one for backwards compatibility,

2019-06-10 11:45:45,995 WARN [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-4] [DefaultRemoteCapabilitiesService] Exception trying to get Applink for manifest with ID 06d0e21c-a66a-308c-a57c-29650c0df38f
2019-06-10 11:45:45,995 INFO [http-nio-8100-exec-11] [AppLinksManifestDownloader] Authenticator placeholder.to.ensure.backwards.compatibility specified by remote application 06d0e21c-a66a-308c-a57c-29650c0df38f is not installed locally, and will not be used.
2019-06-10 11:45:45,996 INFO [http-nio-8100-exec-11] [AppLinksManifestDownloader] Authenticator placeholder.to.ensure.backwards.compatibility specified by remote application 06d0e21c-a66a-308c-a57c-29650c0df38f is not installed locally, and will not be used.
2019-06-10 11:45:55,767 INFO [http-nio-8100-exec-17] [AccessLogFilter] liam PUT https://XXX/bamboo/rest/applinks/3.0/applicationlink 376109kb
2019-06-10 11:45:56,051 INFO [http-nio-8100-exec-18] [AccessLogFilter] liam GET https://XXX/bamboo/rest/applinks/3.0/applicationlink/06d0e21c-a66a-308c-a57c-29650c0df38f?_=1560181543582 371127kb
2019-06-10 11:45:56,076 INFO [http-nio-8100-exec-19] [AccessLogFilter] liam PUT https://XXX/bamboo/rest/applinks/3.0/applicationlink/06d0e21c-a66a-308c-a57c-29650c0df38f/authentication/provider 370226kb
2019-06-10 11:45:56,081 INFO [http-nio-8100-exec-21] [AccessLogFilter] liam PUT https://XXX/bamboo/rest/applinks-oauth/1.0/applicationlink/06d0e21c-a66a-308c-a57c-29650c0df38f/authentication/consumer?autoConfigure=true 369629kb
2019-06-10 11:45:56,123 INFO [http-nio-8100-exec-20] [AccessLogFilter] liam PUT https://XXX/bamboo/rest/applinks/3.0/applicationlink/06d0e21c-a66a-308c-a57c-29650c0df38f/authentication/provider 367846kb

As far as `nonProxyHosts` , nginx should be rewriting the requests to account for this.

At this point I am pretty much at a dead end besides blindly adjusting http.proxy settings. 

Update

Setting nonProxyHost in catalina.properties did not fix the problem:

http.nonProxyHosts=*.atlassian.com|*.atlassian.net|localhost|*.bitbucket.org

Also I found this page and to answer each point:

1. You must have administrator permissions on both the Cloud application (JIRA) and the stand-alone Bamboo.

I Do

1. Bamboo should be accessible via either port 8085, 80 or 443.

It is being served from an alt port, is this really the cause of the 403?

1. If SSL is used, you need to have a valid certificate, a self-signed certificate can NOT be used.

it does

[Daniel Santos]

Hi @upsampled,

Thank you for the detailed description of this question =]

If I understood you correctly you are using a different port, other than 80 or 443. You will need to provide port 80 or 443 for Bamboo.

This is our dor reference:

3. Are you linking Atlassian Cloud applications?

If an application involved in your application link is hosted on Atlassian Cloud, there are a couple of considerations. Your local server must use a valid SSL certificate, and it must be accessible on port 80 or 443. For more information, see this Cloud documentation.

If you have a internet-facing firewall, make sure to allow the IP range used by Atlassian to reach your internal network. For up-to-date information on that, see Atlassian Cloud IP ranges and domains.

These are the steps I would try:

1. Adjust the ports of your proxy so it provides Bamboo with port 80 or 443 to the internet
2. Remove the application links endpoint from both apps
3. Try to connect the application link again from Jira cloud side

In case you still have issues and you have room for testing, please try using an application like NGROK to build a temporary proxy. This will allow you to check if this is a proxy configuration issue or not. Steps for this test:

1. Install https://ngrok.com/download
2. Start a temporary proxy for Bamboo with:

   ngrok http -bind-tls=true <INTERNAL_IP>:<INTERNAL_PORT>

3. Get the ngrok URL.
4. Configure the <Bamboo_Install>/conf/server.xml with the proxy settings below in your Bamboo default connector:

   proxyName="<ngrok_URL>"
   proxyPort="8443"
   scheme="https"

   As described in Securing Bamboo behind nginx using SSL (Step 1: Configure the Tomcat Connector).
5. Start Bamboo and check if you can log in with the ngrok URL.
6. Remove the application links endpoint from both apps.
7. Try to connect the application link again from Jira cloud side.

These set of steps should work and if they do we will be sure the issue is related to the proxy.

Please let us know your findings.

[upsampled]

I have an undesirable working setup right and will wait to post an answer till the support ticket is closed, but I want to correct some documentation right now:

our local server must use a valid SSL certificate, and it must be accessible on port 80 or 443

Bamboo's external IP (the one JIRA connects to) can also be on 8085 via HTTP. The Bamboo instance can be on any port it wants behind the proxy. It also appears that there are restrictions on if you can run HTTPS on anything other than 443. 

[Daniel Santos]

Hi @upsampled

Thank you for pointing that out, I'll file an improvement request for the document that should contain this information. 

This was the page that use to have the whole details about the ports:

• Compare Atlassian cloud vs server

You can see that it was referenced in this community article:

• Connecting Local hosted Crucible to Cloud Jira

The page I mentioned also need a fix:

• Troubleshoot application links - Atlassian Documentation

(...)will wait to post an answer till the support ticket is closed(...)

You mentioned a ticket. Do you have any support ticket open for this problem?