Remote OpenSearch connection-result CONNECTION_ERROR

Lasse Meyer February 21, 2024

I have a Bitbucket Data Center on a Linux VM in Azure and a remote OpenSearch on a Linux VM in Azure, both are in the same network.
OpenSearch is configured like in: Debian - OpenSearch Documentation
And Bitbucket Data Center is configured like in: Install and configure a remote OpenSearch server | Bitbucket Data Center 8.18 | Atlassian Documentation

When i´m using this on my bitbucket data center vm: "curl -X GET https://IP:9200 -u 'user:pw' --insecure" (with my own credentials, which i setuped), i get the right answer from the opensearch server.
But in Bitbucket i still have connection-result: CONNECTION_ERROR

This is the only thing i can find in the atlassian-bitbucket.log and i doesnt get it:


2024-02-21 14:21:05,499 DEBUG [I/O dispatcher 15] c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target



And i dont know why.
Does anyone know or have some ideas?

1 answer

1 vote
Aman Shrivastava
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 21, 2024

Hi Lasse,

The below error indicates that a target application(in this case OpenSearch) is running with an SSL/TLS configuration but it can't be trusted by Bitbucket. This is most likely due to the peer application's certificate being self-issued (i.e., not signed by a CA) or a certificate chain does not exist within the Java trust store that Bitbucket is using.

java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target


I would suggest you review the Unable to connect to SSL services due to "PKIX Path Building Failed" error KB that covers all the possible causes and the fix for this error. In case of any problem, please raise a ticket with Atlassian support.

Regards,
Aman

Lasse Meyer February 22, 2024

Im sorry but this doesnt help me in any way.
Im using self signed certs like the Bitbucket Data Center documentation tells me to do.
But Bitbucket doesnt accept them because their are self signed?

I dont get it.

Aman Shrivastava
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 22, 2024

Hi Lasse,

I will try to clarify all the aspects.

  • The document Install and configure a remote OpenSearch server covers the minimum configuration required to configure the OpenSearch security plugin and in the context of that, it is mentioned that a self-signed certificate can be used if plugins.security.ssl.transport.enforce_hostname_verification is set to false. 
  • It's Java that gives the error when it detects the server's(OpenSearch) certificate can't be trusted. Java maintains a trust store that contains the root and intermediate certificates of public certificate authorities. Based on these certificates, java establishes trust with the server.
  • If you want to use the self-signed certificate then you will need to import the complete certificate chain into the Java trust store which Bitbucket is using. The process to import the certificate in the trust store is included in the KB Unable to connect to SSL services due to "PKIX Path Building Failed" error
     

Regards,
Aman

Like Sabine Mayer likes this
Lasse Meyer February 23, 2024
  1. Check to see that the correct truststore is in use. If -Djavax.net.ssl.trustStore has been configured, it will override the location of the default truststore, which will need to be checked.

    How to do that for bitbucket? I dont find any documenations about it.
    Because, it´s funny it doesnt fail when i check it with SSLPoke..


    root@bitbucketVM2:/bitbucket/jre/bin# ./java -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -Djavax.net.debug=ssl SSLPoke 10.28.11.16 9200
    javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.187 UTC|null:-1|jdk.tls.keyLimits: entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
    javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.207 UTC|null:-1|jdk.tls.keyLimits: entry = ChaCha20-Poly1305 KeyUpdate 2^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472
    javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.484 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
    countdown value = 137438953472
    javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.486 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
    countdown value = 137438953472
    javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.543 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
    countdown value = 137438953472
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.546 UTC|null:-1|No X.509 cert selected for EC
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.546 UTC|null:-1|No X.509 cert selected for EC
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.546 UTC|null:-1|No X.509 cert selected for EC
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.547 UTC|null:-1|No X.509 cert selected for EdDSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.548 UTC|null:-1|No X.509 cert selected for EdDSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.549 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.550 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.550 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.550 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.551 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.552 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.553 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.553 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.554 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.556 UTC|null:-1|No X.509 cert selected for EC
    javax.net.ssl|ALL|10|main|2024-02-23 09:00:26.557 UTC|null:-1|No X.509 cert selected for RSA
    javax.net.ssl|DEBUG|10|main|2024-02-23 09:00:26.565 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
    countdown value = 137438953472
    Successfully connected

Lasse Meyer February 23, 2024

root@bitbucketVM2:/bitbucket/jre/bin# ./java -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -Djavax.net.debug=ssl SSLPoke 10.28.11.16 9200
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.023 UTC|null:-1|jdk.tls.keyLimits: entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.071 UTC|null:-1|jdk.tls.keyLimits: entry = ChaCha20-Poly1305 KeyUpdate 2^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.640 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.644 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.701 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.703 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.713 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.714 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.714 UTC|null:-1|No X.509 cert selected for EdDSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.715 UTC|null:-1|No X.509 cert selected for EdDSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.715 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.715 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.716 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.716 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.717 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.717 UTC|null:-1|No X.509 cert selected for RSASSA-PSS
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.718 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.718 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.718 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.719 UTC|null:-1|No X.509 cert selected for EC
javax.net.ssl|ALL|10|main|2024-02-23 09:06:00.719 UTC|null:-1|No X.509 cert selected for RSA
javax.net.ssl|DEBUG|10|main|2024-02-23 09:06:00.722 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
Successfully connected

Lasse Meyer February 23, 2024

Im not sure what step helped me to get further, but this is my problem now:
Why does it say it doesnt match the CN, when in CN the correct IP is?


2024-02-23 09:35:42,596 DEBUG [I/O dispatcher 5] c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: javax.net.ssl.SSLPeerUnverifiedException: Host name '10.28.11.16' does not match the certificate subject provided by the peer (CN=10.28.11.16, OU=UNIT, O=ORG, L=TORONTO, ST=ONTARIO, C=CA)
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor$1.failed(ApacheRequestExecutor.java:160)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:432)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:82)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:40)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:156)
at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:187)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name '10.28.11.16' does not match the certificate subject provided by the peer (CN=10.28.11.16, OU=UNIT, O=ORG, L=TORONTO, ST=ONTARIO, C=CA)
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.verifySession(SSLIOSessionStrategy.java:217)
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy$1.verify(SSLIOSessionStrategy.java:197)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:372)
at org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:588)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:154)
... 7 common frames omitted

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
VERSION
8.16.0
TAGS
AUG Leaders

Atlassian Community Events