Search - Connection attempt failed to establish connection

[Eric Zimmerman]

Environment

We run a single Bitbucket Data Center server on Ubuntu 22.04 on airgap networks.

We upgraded all of the instances to v8.9.2 about a month ago and all seemed to be well.

Issue

Within the past few days, one of them is exhibiting problem with search.

When I press the "Server settings" "Test" button, the 'atlassian-bitbucket.log', shows:

# note this is an except since I'm manually typing from an airgap server

DEBUG [pool-53-thread-1]  c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: java.net.ConnectException: Connection refused

...

caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)

Notes

• stopped and restarted a number of times
• rebooted the server a number of times
• it seems like search is running on the proper ports
• I ran through a number of things noted here: https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-steps-for-bitbucket-server-code-search-827125334.html 
• Why else might connection be refused?

[Mahesh Shinde]

Hi @Eric Zimmerman :

Have you configured the remote search server or is it bundle with bitbucket application?

Please, validate the below things on your server.

• tail -f $BITBUCKET_HOME/log/search/bitbucket_search.log
  • Nothing happens
• ps -ef | grep search
  • reveals search server is up and running
• netstat -tulnp 
  • as atlbitbucket user does not show the port 7992 on listen. You can't see the PID of your search server listened
• telnet localhost 7992
  • Connection is refused

This will give some clarity where is the exact issue.

Regards,

Mahesh Shinde

[Eric Zimmerman]

re: `tail -f $BITBUCKET_HOME/log/search/bitbucket_search.log`

I see log entries from todays date, ending with `[bitbucket_bundled] Running full sweep`

re: `ps -ef | grep search`

It shows the command line for when it was started yesterday (uses /opt/atlassian/bitbucket/8.9.2/jre/bin/java and opensearch paths look fine.

re: `netstat -tulnp`

I don't have netstat, but the I ran the following `sudo -u atlbitbucket ss -tulnp`.

Ports 7992 and 7993 are displayed as listening with uses "java" and listed pids.

re: `telnet localhost 7992`

`Connected to localhost.`

One other note is that we moved from Windows to Linux last month and the other equivalent server does not exhibit this problem.

[Mahesh Shinde]

Hi @Eric Zimmerman :

As you are using inbuild search server, please access the below URL on browser and provide the credentials to login. 

http://localhost:7992/

If you don't have a local browser on the server that is running Bitbucket Server, you should also be able to run the following command successfully:

$ curl -u username:password http://localhost:7992

[Mahesh Shinde]

If additional troubleshooting is required and the debug log level of the Bitbucket Server to Search server is required, the following command will enable it:

curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" <BASE_URL>/rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug

[Eric Zimmerman]

Thankyou for the feedback.

I found some notes from someone who encountered this before.

Check the configuration file of opensearch. The file name is "opensearch.yml" and it can be found in "<BITBUCKET_HOME>/config/search".

The IP address is determined by "network.host". By default, it may be set to "_local_". If the 7992 port was shown in the netstat results, but the search server test failed, try changing the setting to "network.host = [_local_, _site_]".

Check that the HTTP TCP port is set to 7992.

This ended up solving the issue after I changed this and restarted Bitbucket.

So, my question is why? ... our other servers do not have this set, but search works just fine, so ???