Remote OpenSearch connection-result CONNECTION_ERROR

[Lasse Meyer]

I have a Bitbucket Data Center on a Linux VM in Azure and a remote OpenSearch on a Linux VM in Azure, both are in the same network.
OpenSearch is configured like in: Debian - OpenSearch Documentation
And Bitbucket Data Center is configured like in: Install and configure a remote OpenSearch server | Bitbucket Data Center 8.18 | Atlassian Documentation

When i´m using this on my bitbucket data center vm: "curl -X GET https://IP:9200 -u 'user:pw' --insecure" (with my own credentials, which i setuped), i get the right answer from the opensearch server.
But in Bitbucket i still have connection-result: CONNECTION_ERROR

This is the only thing i can find in the atlassian-bitbucket.log and i doesnt get it:

---

2024-02-21 14:21:05,499 DEBUG [I/O dispatcher 15] c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

---

And i dont know why.
Does anyone know or have some ideas?

[Aman Shrivastava]

Hi Lasse,

The below error indicates that a target application(in this case OpenSearch) is running with an SSL/TLS configuration but it can't be trusted by Bitbucket. This is most likely due to the peer application's certificate being self-issued (i.e., not signed by a CA) or a certificate chain does not exist within the Java trust store that Bitbucket is using.

java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I would suggest you review the Unable to connect to SSL services due to "PKIX Path Building Failed" error KB that covers all the possible causes and the fix for this error. In case of any problem, please raise a ticket with Atlassian support.

Regards,
Aman

[Lasse Meyer]

Im sorry but this doesnt help me in any way.
Im using self signed certs like the Bitbucket Data Center documentation tells me to do.
But Bitbucket doesnt accept them because their are self signed?

I dont get it.

[Aman Shrivastava]

Hi Lasse,

I will try to clarify all the aspects.

• The document Install and configure a remote OpenSearch server covers the minimum configuration required to configure the OpenSearch security plugin and in the context of that, it is mentioned that a self-signed certificate can be used if plugins.security.ssl.transport.enforce_hostname_verification is set to false. 
• It's Java that gives the error when it detects the server's(OpenSearch) certificate can't be trusted. Java maintains a trust store that contains the root and intermediate certificates of public certificate authorities. Based on these certificates, java establishes trust with the server.
• If you want to use the self-signed certificate then you will need to import the complete certificate chain into the Java trust store which Bitbucket is using. The process to import the certificate in the trust store is included in the KB Unable to connect to SSL services due to "PKIX Path Building Failed" error
   

Regards,
Aman

[Lasse Meyer]

1. Check to see that the correct truststore is in use. If -Djavax.net.ssl.trustStore has been configured, it will override the location of the default truststore, which will need to be checked.
   
   How to do that for bitbucket? I dont find any documenations about it.
   Because, it´s funny it doesnt fail when i check it with SSLPoke..
   
   

   ---

   root@bitbucketVM2:/bitbucket/jre/bin# ./java -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -Djavax.net.debug=ssl SSLPoke 10.28.11.16 9200
   javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.187 UTC|null:-1|jdk.tls.keyLimits: entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
   javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.207 UTC|null:-1|jdk.tls.keyLimits: entry = ChaCha20-Poly1305 KeyUpdate 2^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472
   javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.484 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
   countdown value = 137438953472
   javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.486 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
   countdown value = 137438953472
   javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.543 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
   countdown value = 137438953472
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.546 UTC|null:-1|No X.509 cert selected for EC
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.546 UTC|null:-1|No X.509 cert selected for EC
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.546 UTC|null:-1|No X.509 cert selected for EC
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.547 UTC|null:-1|No X.509 cert selected for EdDSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.548 UTC|null:-1|No X.509 cert selected for EdDSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.549 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.550 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.550 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.550 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.551 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.552 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.553 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.553 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.554 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.556 UTC|null:-1|No X.509 cert selected for EC
   javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.557 UTC|null:-1|No X.509 cert selected for RSA
   javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.565 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
   countdown value = 137438953472
   Successfully connected
   
   

[Lasse Meyer]

root@bitbucketVM2:/bitbucket/jre/bin# ./java -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -Djavax.net.debug=ssl SSLPoke 10.28.11.16 9200
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.023 UTC|null:-1|jdk.tls.keyLimits: entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.071 UTC|null:-1|jdk.tls.keyLimits: entry = ChaCha20-Poly1305 KeyUpdate 2^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.640 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.644 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.701 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.703 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.713 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.714 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.714 UTC|null:-1|No X.509 cert selected for EdDSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.715 UTC|null:-1|No X.509 cert selected for EdDSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.715 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.715 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.716 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.716 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.717 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.717 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.718 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.718 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.718 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.719 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.719 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.722 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
Successfully connected

[Lasse Meyer]

Im not sure what step helped me to get further, but this is my problem now:
Why does it say it doesnt match the CN, when in CN the correct IP is?

---

2024-02-23 09:35:42,596 DEBUG [I/O dispatcher 5] c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: javax.net.ssl.SSLPeerUnverifiedException: Host name '10.28.11.16' does not match the certificate subject provided by the peer (CN=10.28.11.16, OU=UNIT, O=ORG, L=TORONTO, ST=ONTARIO, C=CA)
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor$1.failed(ApacheRequestExecutor.java:160)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:432)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:82)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:40)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:156)
at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:187)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name '10.28.11.16' does not match the certificate subject provided by the peer (CN=10.28.11.16, OU=UNIT, O=ORG, L=TORONTO, ST=ONTARIO, C=CA)
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.verifySession(SSLIOSessionStrategy.java:217)
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy$1.verify(SSLIOSessionStrategy.java:197)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:372)
at org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:588)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:154)
... 7 common frames omitted