jira health check of gadget problem

[James Jung]

Dear who may concern,

I tried to access my team's JIRA using HTTPS,

After following every steps described in atlassian official documents,

I failed on gadget health check

I used apache2.2 as my reverse proxy to jira at localhost:8080

and rewrite module to force http redirect to https,

we used paid SSL certificates for sure,

also I definitely followed the passage of "How can I resolve this?"

and adjusted settings.

Here is the support zip file and some settings' snapshots.

Hoping to get some solutions from your support.

Thank you!

support.zip

 

 

[James Jung]

Dear Andrew,

Thanks again for your warm replies,

I did the SSLPoke and the result looked ok:

however, I found my problem and resolved by removed the arguments listed in this article:

https://confluence.atlassian.com/jirakb/how-to-configure-an-outbound-http-and-https-proxy-for-jira-applications-247857187.html

it told readers to add :

-Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=localhost

 in setenv.sh's "JVM_SUPPORT_RECOMMENDED_ARGS"

but the arguments are conflicted to each other,

then I accidentally removed the "http.proxyHost=","http.proxyPort=" and the "https" ARGS as well, only left the "http.nonProxyHosts=localhost" and "https.nonProxyHosts=localhost" ARGS,

the gadget problems amazingly solved,

no more "_MSG_gadget" titles!

Unfortunately,  my application link still unreachable,

somehow, the jira cannot connect outbound to another applications anymore,

every time I tried to establishing application link, it shows the error messages as below,

my confluence uses the same way to redirect to https through SSL,

and it works fine on https://confluence.sso.edu.tw

still need supports,

Sincerely, 

[Andy Heinzer]

Hi James,

Glad to hear that the first part of the problem seems resolved.  But I understand that the application links are still not working as expected.

Is 'con1' a context path that confluence is running?   If so, then the question is how is the proxy for Confluence setup?  Many times when Jira or Confluence is setup to use a context path and a reverse proxy, the proxy can rewrite the requests so that the end users are unaware of this.  It can help to make the URL shorter and more concise.

Another way to try to troubleshoot this is to try to bypass both the proxy and the use of SSL by Creating an unproxied application link per that guide.  Try this and let me know the results.

Andy

[James Jung]

Thank you, Andy,

After few days digging, I found that our higher level network management authority set the policies which only allow port80 & 443 traffics to passing through main firewall and load balance mechanism, so I cannot directly bypass to connect my jira instance at port8080 and I don't have the vpn permission either, sadly to say,

Is there any other way to test application link?

Or is it related with apache rewirte module? I do use rewrite rule and redirect in apache.

Sincerely in-need of supports,

[Andy Heinzer]

Creating an unproxied application link is a good way to bypass these things like load balancers and proxies in order to be able to still connect these applications with each other.    Even if you do not want to use this method in the long term, it can still be helpful as a troubleshooting measure, just to better understand where the problem exists in creating application links between these two programs.

Are Jira and Confluence both behind this same firewall?  If so, then you don't actually have to create new rules on the firewall to make this work.   The configuration of both Jira and Confluence control their respective tomcat web servers via their $installdir/conf/server.xml files.  It is possible that both Jira and confluence could be accessed on more than one port at a time.   The guide I recommended explains this and provides instructions to follow to make this work.

So while the public could reach your jira site on the full public URL on port 80 or 443, you could still create a new port number as mentioned in that guide.  Then you could use an internal address for creating this application link.  This would be something more closely to the private IP address and port number.   This method can be used when both Jira and Confluence sit behind the same firewall AND are able to communicate with each other.   If Jira and confluence are on the same address, you can likely just use the localhost address in regards to creating this unproxied application link.  But even if they are on separate hosts, having them on the same private network can help here.  This way the applications could still communicate with each other without having to use the public address or navigate the possible configuration changes to your load balancer/firewall.

If your Jira and confluence are not on the same private network, then I think it would be beneficial to either relocate the application so they are on the same network.

[James Jung]

Dear @Andy Heinzer,

Thank you for your clear and detailed description, I think that is better than those paragraph on JIRA's official documents,

after I recognized my network setups, I did the bypassing link successfully, and that credit is yours, I'm grateful.

  

however, when I tried to synchronizing my confluence's user directory to JIRA, it failed,

I did setup to sharing user directory with JIRA when I installed my confluence, and it did synchronize successfully at the beginning when there was no proxy or ssl in between,

but now this fail message breaks my tiny hope of success,

error message shows as following:

The following URL does not specify a valid Crowd User Management REST service: http://jira.sso.edu.tw/jira/rest/usermanagement/1/search?entity-type=user&start-index=0&max-results=1&expand=user

Is there any thing that I can do to fix this?

Sincerely,

[Andy Heinzer]

Hi James,

The fact that you see both that error, and that the sync is failing, tends to indicate to me that your confluence installation cannot communicate with that specific address.   I suspect you can still login because there is some cached information in confluence that allows this.

However there is a Jira KB that might help here: How to troubleshoot The following URL does not specify a valid Crowd User Management REST service.

I suspect that because the application is trying to reach the fqdn address ( http://jira.sso.edu.tw/jira/ ), and I think we have established that you have a proxy here that might be blocking such requests, I think you might need to adjust the settings for this connect to use an alternative internal address instead.

Again this would be something like a private ip address that is only accessible between Jira/Crowd and your Confluence instance, just like you did for the creation of the unproxied application links.   The problem with doing this is that you can't change that setting in confluence when you are logged in through an account on that directory.   So instead, you will likely want to login as a Confluence Administrator account that exists only in the Internal user directory.  If you can do that, then you should be able to edit this user directory in Confluence to use an alternative address to reach this Jira/Crowd user directory.

[Martin Zeller]

@James Jung

If you can only reach 80 / 443 and your application ports are on a higer blocked port, then you only have to point the proxy to 80 / 443 - no redirect rules.

The redirect can happen through the firewall.

for example (here with zone=internal)

sudo firewall-cmd --permanent --zone=internal --add-forward-port=port=80:proto=tcp:toport=8080
sudo firewall-cmd --permanent --zone=internal --add-forward-port=port=443:proto=tcp:toport=8443

 hope that helps ;)

Cheers Martin

[Andy Heinzer]

Hi James,

Thanks for posting the support zip.  I took a look at this configuration and I think I see a possible problem that might be causing this health check to throw a warning.

In your $JIRAINSTALL/conf/server.xml file you have defined two connectors for Jira's Tomcat server to host the site from: 

1. One which uses port 8080 and
2. The other that uses 8443 along with the SSL settings defined.

The problem here is that since you appear to doing port forwarding and accessing the site on that https://yourdomain this means that your end users are connecting on port 443.  But Jira isn't technically hosting it's site on that port.  Because there exists this redirection, we need to let Tomcat know about this redirection.

What I would recommend would be to add the following to your connector running on port 8443:

        proxyName="jira.sso.edu.tw"
        proxyPort="443"
        scheme="https"

I see you already have that for the 8080 connector, but since the traffic to Jira ultimately has to go through the 8443 connector, you will need those parameters there as well.  Otherwise Jira will not really understand where this traffic is coming from and in turn can cause this error and have other problems with rendering content.

Then save that file and restart JIra once more.   Try that and see if you still get this health check warning when you next start up Jira.

[James Jung]

Dear Andrew,

Thanks for your efforts,

however, I got no luck on that,

I added your suggestion into my 8443 connector as following:

and rebuild the jks as well, then restart jira, still got same health check warning,

I went to view the catalina.out log,

it shown the warning as below:

WARNING [http-nio-8080-exec-19] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://jira.sso.edu.tw/jira/rest/gadget/1.0/login, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

Is there any suggestions to solve this issue?

Please advise me.

[Andy Heinzer]

Hi James,

In your first post you cited a link to this KB: Health Check: JIRA Base URL via the "How can I resolve this?" link.

While I think this is related to your problem, I just wanted to understand more about what specific behaviors your Jira instance is exhibiting.  For example, when you look at the dashboard of Jira that contains gadgets, are you seeing the titles for those gadgets?  Or are you seeing a title like '__MSG_gadget.activity'

This will tell us more about the problem at hand.

I have seen instances that can sometimes throw this health check error incorrectly, in those cases the gadgets all display their titles correctly.  However if the gadgets are also showing this incorrect title, then it seems likely there is still a problem there that could be resolved by that KB.   This KB explains that these problems happen when the Jira application is not able to reach the base URL address set for the Jira instance.  That can happen for a number of different reasons explained in that KB.

[James Jung]

Gratefully thank you again, Andrew,

I've been working on this situation for more than few days,

and yes, you're right about the gadget title,

it does shows '__MSG_gadget.activity'

and my situation is behind an Apache2.2 reverse proxy,

I tried every job described in this article: 

https://confluence.atlassian.com/jirakb/how-to-fix-gadget-titles-showing-as-__msg_gadget-813697086.html

except with this one:

 

I can't add this to my white lists through my web UI,

is there any connection with my problem?

Really need some supports,

Sincerely,

[Andy Heinzer]

Before you jump to the resolution steps, with that KB, it is really more important to walk through the diagnosis steps in order to better understand why you are getting this error.

The general concept behind why you see this error is that the Jira Application cannot connect to it's own base URL.  But there are lots of different possible causes for that.

Specifically, I would recommend trying these three tests from How to fix gadget titles showing as __MSG_gadget and let us know the results of each of these:

• From the JIRA server itself, JIRA is unable to communicate with itself via its Base URL (Run curl -v <base_url> to verify)
• Run SSLPoke from the JIRA server itself and see if it returns successfully
• Additionally run the httpclienttest from the JIRA server itself to confirm if the SSL configuration is okay, as this will verify if you're affected by  JRASERVER-47568 - JIRA complains about SNI host Resolved

The last one might not be as important.  However if we can learn the results of the curl and the SSLPoke (when done from the Jira server itself), these results should be able to tell us more about what the problem is with this specific configuration.  From there I should be able to offer more specific troubleshooting steps to help resolve this.