Schannel 36887 - The TLS protocol defined fatal. Alert code 46

[John Harmon]

I have configured Jira for ldap over 636, and imported our ca certs into the keystore.  While everything appears to work from Jira's side of things, from the AD side we are seeing this error:

Schannel 36887 - A fatal alert was received from the remote endpoint. The TLS protocol defined fatal alert code is 46

More on this error code:
https://blogs.msdn.microsoft.com/kaushal/2012/10/05/ssltls-alert-protocol-the-alert-codes/

This seems to work fine with confluence (same certs).  I cannot, for the life of me, figure out what to do with this one.  I have re-imported certs (multiple ways), converted from JKS to PKCS12 (and back again), and everything else I can think of.

I could sure use some pointers.

Encrypted LAN trace shows Jira request some info, AD hands it back with a cert, and we reject it as unknown or mangled.  Connection is then reset.  Again, everything appears to be working fine symptomatically, but it is bugging our AD admin to no end.

[Andy Heinzer]

Are Jira and Confluence using the same keystore and the same Java runtime?   I ask because to see this problem in one application, but not the other to me suggests that the two applications are using different Java runtimes, that have different keystore locations.  It's not uncommon to see this, since both Jira and Confluence have installation packages that do bundle an Oracle JRE package.

If you're OS has a $JAVA_HOME (linux/unix) or %JAVA_HOME% (windows) variable set, then this should help unify and clarify exactly which java path is being used by each application.  If you haven't done this yet, or are not sure, I'd recommend walking through Installing Java for steps on how to do this.  If you have to set this path, I'd recommend then restarting Jira and repeating the steps to import the certificate again to the new keystore path.

If you're still having this problem after those steps, take a look at this thread: https://serverfault.com/questions/639837/openssl-s-client-shows-alert-certificate-unknown-but-all-server-certificates-app

The error code 46 of that indicates 'certificate_unknown' and there is some unspecified problem processing that cert.   In that thread it appears that at least one user found they also needed to import the root CA cert in order to make their cert work.

[John Harmon]

JIRA and CONFLUENCE reside on different servers, so they are using different keystore files; however, I did configure them at the same time, so the only difference between the keystore files is:

1. The name of the keystore
2. Location of the keystore
3. Confluence has a cert for Jira in it, and vice~versa (for application-links)

The Java runtime being utilized is the one that ships with each application.  I didn't install an external one.

• JIRA: JAVA_HOME="/opt/atlassian/jira/jre/"
• CONFLUENCE: JRE_HOME="/opt/atlassian/confluence/jre/"

java -version reports (for both servers):

openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

There is a symlink for java in the PATH (linux) found under /usr/bin/java, which is a symlink to /etc/alternatives/java, which is a symlink to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/bin/java

This was setup by the installer on both systems.

[Andy Heinzer]

Thanks for that information.   If each application is using the bundled JRE, then you should be okay. However I should note that neither Jira or Confluence actually support OpenJDK at this time.  Both only support the official Oracle Java JRE/JDK.  I don't think that is a causing factor here since both are using the bundled JREs, but just wanted to clarify.

I would be interested to see if you can use the SSLPoke utility mentioned in the KB Unable to Connect to SSL Services due to PKIX Path Building Failed.  I don't have any reason to believe that you are seeing the same error as that KB, however the SSLPoke utility found on that page can help us understand more about your environment.   This utility is a java class you can run and then pass it address and port parameters just to test if your java environment can establish a connection to this other SSL resource.   Even if this doesn't work, then the results of the test should tell us more about this problem.  

In your case, we are going to want to make sure we're running this with the same java version that Jira is using.   Otherwise we're not really getting a fair comparison. 

[John Harmon]

Thank you for your help.

I ran the SSLPoke on both JIRA and CONFLUENCE servers, and they both reported "Successfully connected"

ie.

# /opt/atlassian/jira/jre/bin/java SSLPoke dc1.mydomain.com 636
Successfully connected

I then reran each SSLPoke command with the following (similar on Confluence), pointing each server to their own java key store, and those commands too reported "Successfully connected"

# /opt/atlassian/jira/jre/bin/java -Djavax.net.ssl.trustStore=/var/atlassian/application-data/jira/jira.jks SSLPoke dc1.mydomain.com 636
Successfully connected

[John Harmon]

I wonder if I did something wrong with the latter.  I say that because I would have expected it to prompt me for the keystore password, but it didn't.  Perhaps I don't understand how the command works (ie password not needed for the keystore)

[John Harmon]

Still no progress.  I changed the sync interval to monthly instead of hourly (lightly used system) so it isn't as annoying to the AD admin, but I am at a loss as to what to do.

[Andy Heinzer]

Hi John,

I did some more research on this issue, and I came across this link:

https://social.technet.microsoft.com/Forums/en-US/e262fca9-da70-482b-85b6-c39a0ea087e2/schannel-fatal-alert-46?forum=exchangesvrsecuremessaging

Which in turn lead me to this one: https://practical365.com/exchange-server/configuring-the-tls-certificate-name-for-exchange-server-receive-connectors/

It appears to indicate that you might actually have to configure some additional settings on the server side that indicate the client host/username being used in this kind of setup.  I hope this helps.

Andy