Unable to add Application Link

[Juan Carlos Crisostomo]

We have encountered an issue when attempting to add an Application Link between Jira and Confluence. Both systems have been secured with SSL, they shared the same wildcard SSL certificate issued by Starfield. I have added the public certs for confluence to the keystore  under <jira dir>/jre/lib/security/cacerts. I am able to list and see them. However, I am unable to reach the site. I have tried getting the manifest manually using the CURL command (similarly added the public cert to the keystore used by curl) and it work just fine. 

The following was logged when attempting to add the application link:

2017-08-18 08:19:38,229 https-jsse-nio-9757-exec-14 ERROR creacore 499x222x2 141yj45 17.5.7.50 /rest/applinks/3.0/applicationlinkForm/manifest.json [c.a.a.c.rest.ui.CreateApplicationLinkUIResource] ManifestNotFoundException thrown while retrieving manifest
com.atlassian.applinks.spi.manifest.ManifestNotFoundException: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.doDownload(AppLinksManifestDownloader.java:198)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.access$000(AppLinksManifestDownloader.java:50)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader$1$1.<init>(AppLinksManifestDownloader.java:127)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader$1.load(AppLinksManifestDownloader.java:121)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader$1.load(AppLinksManifestDownloader.java:118)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.downloadInternal(AppLinksManifestDownloader.java:106)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.download(AppLinksManifestDownloader.java:84)
at com.atlassian.applinks.core.manifest.ManifestRetrieverDispatcher.getManifest(ManifestRetrieverDispatcher.java:43)
at com.atlassian.applinks.core.rest.ui.CreateApplicationLinkUIResource.tryToFetchManifest(CreateApplicationLinkUIResource.java:140)
... 3 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
... 3 filtered
at com.atlassian.applinks.internal.rest.interceptor.NoCacheHeaderInterceptor.intercept(NoCacheHeaderInterceptor.java:13)
... 1 filtered
at com.atlassian.applinks.core.rest.auth.AdminApplicationLinksInterceptor.intercept(AdminApplicationLinksInterceptor.java:35)
... 1 filtered
at com.atlassian.applinks.core.rest.context.ContextInterceptor.intercept(ContextInterceptor.java:16)
... 15 filtered
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:154)
... 1 filtered
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:68)
... 36 filtered
at com.atlassian.applinks.cors.rest.CorsFilter.doFilter(CorsFilter.java:99)
... 3 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 53 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:80)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:78)
... 16 filtered
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)
... 20 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 10 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 29 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 29 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:248)
at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:270)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:103)
at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:131)
at com.atlassian.plugins.rest.module.jersey.JerseyRequest.execute(JerseyRequest.java:113)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.doDownload(AppLinksManifestDownloader.java:165)
... 260 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1617)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:908)
at com.atlassian.sal.core.net.HttpClientRequestFactory.createHttpClient(HttpClientRequestFactory.java:47)
at com.atlassian.sal.core.net.HttpClientRequestFactory.createRequest(HttpClientRequestFactory.java:40)
at com.atlassian.sal.core.net.HttpClientRequestFactory.createRequest(HttpClientRequestFactory.java:29)
... 3 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy1043.createRequest(Unknown Source)
at com.atlassian.plugins.rest.module.jersey.JerseyRequestFactory.createRequest(JerseyRequestFactory.java:31)
... 3 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy1346.createRequest(Unknown Source)
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.doDownload(AppLinksManifestDownloader.java:161)
... 260 more
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
at sun.security.util.DerInputStream.getLength(DerInputStream.java:561)
at sun.security.util.DerValue.init(DerValue.java:365)
at sun.security.util.DerValue.<init>(DerValue.java:320)
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1914)
at java.security.KeyStore.load(KeyStore.java:1445)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultTrustManager(SSLContextImpl.java:767)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:733)
... 3 filtered
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.security.Provider$Service.newInstance(Provider.java:1595)
... 313 more

Any assitance will be greatly appreciated.

Thanks,

[Branden McElveen]

Hi Juan,

The first "caused by" is the following:

Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.

at sun.security.util.DerInputStream.getLength

In light of that I would start with the article titled Java Certificate Issue - IOException: DerInputStream.getLength(): lengthTag=109, too big.

Here are some possible causes for this:

• The keystore configured for SSL couldn't be read by keytool neither as JKS nor PKCS#12 keystore. So, this can affect Tomcat.
• Often how this error occurs is when certs are copied from Windows to Unix ie you have raised a CSR and you have received via email a signed certificate or a set of multiple signed certs from your certificate Authority. A CA could be Verisign or other main stream vendor or even the security team within your organization.
• There are extra characters found at the end of the certificate file which the “certificate parser” is attempting to interpret as the start or end of a certificate section. The most common way to encounter this error is to have one, or more, blank lines at the end of the certificate file.  A line termination sequence is permitted (but not required) at the end of the final “—–END”  line (Sometimes you may have more than one encoded cert in a file), but there can be no more than one termination sequence of characters. Reference to webspheretools.com.

Please take a look at Java Certificate Issue - IOException: DerInputStream.getLength(): lengthTag=109, too big for additional diagnostic information and resolution steps.

Cheers,

Branden

[Juan Carlos Crisostomo]

You were on the right track Branden.

First Problem:
The keystore was type JKS as per SSL Certificate provider and Atlassian recommendations (CSR was generated for Tomcat). I followed the steps provided and the keystore was verified under a diferent server, so the integrity of the keystore was not the issue. However, I needed to make sure to setup the SSL connection correctly by specifying the keystoreType or the keyAlias under server.xml connection declaration. Here is a sample:

<Connector port="portnum" protocol="org.apache.coyote.http11.Http11NioProtocol"
 maxHttpHeaderSize="8192" SSLEnabled="true"
 maxThreads="150" minSpareThreads="25"
 enableLookups="false" disableUploadTimeout="true"
 acceptCount="100" scheme="https" secure="true"
 clientAuth="false" sslProtocol="TLS" useBodyEncodingForURI="true"
 keystoreType="JKS" keystorePass="****" keyAlias="tomcat"
 keystoreFile="<file location>"/>

Thanks for the help.

JC

[Juan Carlos Crisostomo]

Second problem:
Once the SSL certificate was properly recognized, I still had issues connecting to the Confluence and Bitbucket as I needed to add the Public Cert for my wildcard SSL Certificate to the default JRE keystore "cacerts". If you are using the internal JRE you must add the cert to:

<JIRA APPLICATION PATH>/jre/lib/security/cacerts.

or, JRE on your Java install /usr/lib/jvm/<java version>/jre/lib/security/cacerts.

In order to obtain the public cert and add it to the keystore please use the following steps:

~$ sudo openssl s_client -connect <SSL Site Domain>:<PORT> < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > <CERT_DOMAIN>.cert

where:

<SSL Site Domain> - Working site secured with SSL Cert. Example: bitbucket.mydomain.com

<Port> - SSL Port, usually 443

~$sudo <JAVA_HOME_DIR>keytool -import -alias sslcertalias -keystore <JAVA_HOME_DIR>/lib/security/cacerts -file <CERT_DOMAIN>.cert

<JAVA_HOME_DIR> - Java JRE Home directory

Once I did this, all issues were resolved.
Thanks for the help.
JC