Search - Connection attempt failed to establish connection

Question

Environment

We run a single Bitbucket Data Center server on Ubuntu 22.04 on airgap networks.

We upgraded all of the instances to v8.9.2 about a month ago and all seemed to be well.

Issue

Within the past few days, one of them is exhibiting problem with search.

When I press the "Server settings" "Test" button, the 'atlassian-bitbucket.log', shows:

# note this is an except since I'm manually typing from an airgap server

DEBUG [pool-53-thread-1]  c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: java.net.ConnectException: Connection refused

...

caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)

Notes

* stopped and restarted a number of times
* rebooted the server a number of times
* it seems like search is running on the proper ports
* I ran through a number of things noted here: https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-steps-for-bitbucket-server-code-search-827125334.html 
* Why else might connection be refused?

Eric Zimmerman_1zzska · Answer

Thankyou for the feedback.

I found some notes from someone who encountered this before.

Check the configuration file of opensearch. The file name is "opensearch.yml" and it can be found in "<BITBUCKET_HOME>/config/search".

The IP address is determined by "network.host". By default, it may be set to "_local_". If the 7992 port was shown in the netstat results, but the search server test failed, try changing the setting to "network.host = [_local_, _site_]".

Check that the HTTP TCP port is set to 7992.

This ended up solving the issue after I changed this and restarted Bitbucket.

So, my question is why? ... our other servers do not have this set, but search works just fine, so ???

Mahesh Shinde_k4hefc · Answer

If additional troubleshooting is required and the debug log level of the Bitbucket Server to Search server is required, the following command will enable it: curl -u -v -X PUT -d "" -H "Content-Type: application/json" /rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug

Mahesh Shinde_k4hefc · Answer

Hi @Eric Zimmerman : As you are using inbuild search server, please access the below URL on browser and provide the credentials to login. http://localhost:7992/ If you don't have a local browser on the server that is running Bitbucket Server, you should also be able to run the following command successfully: $ curl -u username:password http://localhost:7992