Application Health fails even it says 'connected'

[mharig]

Hi,

i have running Confluence 5.9.1 and JIRA 7.4.1 on the same server. There is a running nginx as reverse Proxy (configured as suggested by atlassian docu). Both applications are using the same wildcard certs which is trusted by the applications (imported in the corresponded truststore).
After creating an application link it is added without any problems. In Jira it says 'Connected' and the connection works in both directions.

Now in the health check JIRA says that there is a problem with the application link (it is the only one at the moment). 

Debugoutput for 'com.atlassian.applinks':

2017-07-17 17:33:25,470 HealthCheck:thread-5 DEBUG      [c.a.a.core.auth.ApplicationLinkRequestFactoryFactoryImpl] Cannot create request factory with authentication provider 'com.atlassian.applinks.api.auth.types.OAuthAuthenticationProvider' without current user context. 
2017-07-17 17:33:25,472 HealthCheck:thread-5 WARN      [c.a.applinks.core.DefaultApplinkStatusService] Unrecognized error while attempting to retrieve status of Application Link 'UUID-UUID-UUID' 
2017-07-17 17:33:25,472 HealthCheck:thread-5 DEBUG      [c.a.applinks.core.DefaultApplinkStatusService] Unrecognized error trace for 'UUID-UUID-UUID' 
java.lang.NullPointerException 
        at com.atlassian.applinks.internal.rest.client.RestRequestBuilder.buildRequest(RestRequestBuilder.java:251) 
        at com.atlassian.applinks.internal.rest.client.RestRequestBuilder.build(RestRequestBuilder.java:226) 
        at com.atlassian.applinks.internal.status.oauth.remote.ApplinkAuthenticationOAuthFetchStrategy.fetch(ApplinkAuthenticationOAuthFetchStrategy.java:72) 
        at com.atlassian.applinks.internal.status.oauth.remote.DefaultRemoteOAuthStatusService.fetchInternal(DefaultRemoteOAuthStatusService.java:97) 
        at com.atlassian.applinks.internal.status.oauth.remote.DefaultRemoteOAuthStatusService.fetchOAuthStatus(DefaultRemoteOAuthStatusService.java:84) 
        at com.atlassian.applinks.core.DefaultApplinkStatusService.getApplinkStatus(DefaultApplinkStatusService.java:97) 
        at com.atlassian.support.healthcheck.impl.JiraApplinksStatusService.lambda$null$0(JiraApplinksStatusService.java:52) 
        at com.atlassian.applinks.core.ElevatedPermissionsServiceImpl.executeAs(ElevatedPermissionsServiceImpl.java:31) 
        at com.atlassian.support.healthcheck.impl.JiraApplinksStatusService.lambda$null$3(JiraApplinksStatusService.java:51) 
        at java.util.Optional.map(Optional.java:215) 
        at com.atlassian.support.healthcheck.impl.JiraApplinksStatusService.lambda$getStatus$4(JiraApplinksStatusService.java:45) 
        at java.util.Optional.map(Optional.java:215) 
        at com.atlassian.support.healthcheck.impl.JiraApplinksStatusService.getStatus(JiraApplinksStatusService.java:44) 
        at com.atlassian.support.healthcheck.checks.ApplinksStatusHealthCheck.check(ApplinksStatusHealthCheck.java:33) 
        at com.atlassian.support.healthcheck.impl.PluginSuppliedSupportHealthCheck.check(PluginSuppliedSupportHealthCheck.java:51) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Any advice?

Best regards,

Marcel 

[vboerner]

After upgrading to Jira 7.11 with Confluence 6.10 I see the same message sometimes. I can't reproduce this.
Catalina.out contains, e.g.:

HealthCheck:thread-4 WARN      [c.a.applinks.core.DefaultApplinkStatusService] Unrecognized error while attempting to retrieve status of Application Link 'd1e99889-0     0f6-3461-8208-aa2e5c016a66'

- OR -

HealthCheck:thread-6 WARN [c.a.applinks.core.DefaultApplinkStatusService] Unrecognized error while attempting to retrieve status of Application Link 'd1e99889-0 0f6-3461-8208-aa2e5c016a66'

The entry occurs every 3 hours next to the ServiceRunner logs

The last timestamp was 09:48:55, and ad 09:48:10 I see an entry in the Jira-Webinterface > "Scheduler Details" (unfortunately without a message):

CompatibilityPluginScheduler.JobId.hipchatInstallGlancesJob

Type  Runnable

Parameters  {}

Run mode  once per cluster

Schedule  1 hour

Last Run  Fri Aug 03 09:48:10 CEST 2018

Last run duration  13 milliseconds

Next Run  Fri Aug 03 10:48:10 CEST 2018

Message

Deleting the Application-Link on the Jira-site and setting it up again did NOT help.

2018-08-03 13:48:55,802 HealthCheck:thread-6 WARN [c.a.applinks.core.DefaultApplinkStatusService] Unrecognized error while attempting to retrieve status of Application Link 'd1e99889-00f6-3461-8208-aa2e5c016a66'
2018-08-03 13:48:55,802 HealthCheck:thread-6 DEBUG [c.a.applinks.core.DefaultApplinkStatusService] Unrecognized error trace for 'd1e99889-00f6-3461-8208-aa2e5c016a66'

 ... so I searched in the catalina.out for the ApplicationLink-ID from this message ( meaning: "d1e99889..." in my case)

There I found another error (shortened):

2018-08-03 03:33:42,425 http-nio-8080-exec-21 WARN anonymous 213x313x1 [......] /rest/applinks/latest/applicationlink/d1e99889-00f6-3461-8208-aa2e5c016a66/authentication/provider ....... Problem encountered authentication OAUTH ........ 
<<<<<<...I resume that line on the next line: >>>>>>>>
...., error was "consumer_key_unknown", with parameters "{oauth=consumer_key_unknown}"

 ... that led me to this thread: https://confluence.atlassian.com/kb/oauth-error-oauth_problem-consumer_key_unknown-720406729.html

So I looked into the jira database, but there is no table with the name "bandana".
I had to look into the confluence-database, and there is indeed a duplicate entry in the "bandana" table:
 bandanaid
----------
   1638435
   1638434

I cant shut down JIRA at the moment, but maybe following that thread could solve the problem

[Allison Kellner]

I would be interested to know if there is any additional feedback on the above. We are now seeing similar behavior after upgrading JIRA from 7.1.6 to v7.4.5. We are running on Confluence 6.2.4. We also have Bitbucket, Fisheye/Crucible linked to JIRA as well. We are seeing the Health Check application link error however when checking the application links all are connected with no errors.

I have turned on DEBUG logging to see if the error returns and I can capture what is causing the issue. But if any other details are available prior to then it would be much appreciated!

[somethingblue]

Hi Marcel,

From the information you provided it appears that is a symptom of something else.  Here's what I would do:

1. Enable debug logging (or keep it on if you haven't turned it off)
2. Delete the Application link
3. Re-create the application link

Check the logs for any errors during the process of creating the applicaiton link such as a PKIX error, etc. 

Cheers,

Branden

[mharig]

Hi Branden,

thank you. Unfortunately I don't see any errors during the process.

But I find something interesting:

After the 401 JIRA seems to try it again with a user_context  and then the connection is successfull.

Do you need need further input for better understanding my issue/problem?

Thank you in advanced,

Marcel

Edit:
I wanted to post the log output but it says 

Error

Your reply was not accepted. Please check for inappropriate language or spam.

So i added it as screenshot

[mharig]

Hey @somethingblue 

Do you have any updates/suggestions/advice for me?

Thank you in advanced,

Marcel