Elasticsearch not starting automatically with Bitbucket 5.5.0 (Linux)?

[OneTech IT]

We have Bitbucket 5.5.0 running under a dedicated user account (atlbitbcket) on Linux (CentOS 7.x).  We now have need to have Elasticsearch running on the server (I can see in the logs it doesn't start up automatically when Bitbucket does) and when I try to go to http://myserver.mydomain.com:7992 it doesn't respond.

I also noticed the following in System Information:

base-url: http://localhost:7992/
username: bitbucket
hasPassword: true
connection-result: AUTHENTICATION_ERROR

So it appears that Elastic Search is trying to start using the user 'bitbucket' instead of 'atlbitbucket'.   And there is no user 'bitbucket' defined on my server.

Where can I change these config settings, to have Elasticsearch service attempt to start, by using the user 'atlbitbucket' (with no password), to see if that is the issue?

How do I enable Elasticsearch and how do I get it to run under the same non-privileged user account that my Bitbucket server does?

[Kanwar]

Hi OneTech IT,

The error indicates that username/password combination to connect to elasticsearch may be incorrect.

There are troubleshooting steps on this page: Check the username/password for Elasticsearch

Summary of the troubleshooting steps on the above linked page:

Locate the $BITBUCKET_HOME/shared/search/buckler/buckler.yml file note the username/password and update these values on the "Server Settings" page of Bitbucket Server GUI. You can test the connection by clicking the 'Test' button on the page. Once done, elasticsearch should be operational.

Alternatively, you can also configure elasticsearch in bitbucket.properties:

• plugin.search.elasticsearch.baseurl
• plugin.search.elasticsearch.username
• plugin.search.elasticsearch.password

Restart Bitbucket and elasticsearch should work.

Let us know if this helps.

Kanwar

[Matthew Pease]

I'm facing a similar issue. I checked buckler.yml for the default login and copied to the GUI but when I click the Test button, it fails. The log indicates the connection to ElasticSearch is refused with such entries as:

./atlassian-bitbucket.log:2018-05-16 13:52:49,487 DEBUG [Caesium-1-3]  c.a.b.i.s.c.c.DefaultElasticsearchClientFactory Create new inspecting Elasticsearch client using server URL 'http://localhost:7992/'

./atlassian-bitbucket.log:2018-05-16 13:52:49,492 INFO  [pool-66-thread-1]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed due to exception java.net.ConnectException: Connection refused

./atlassian-bitbucket.log:2018-05-16 13:52:49,493 DEBUG [pool-66-thread-1]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Detailed Exception: 

./atlassian-bitbucket.log: at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor$1.failed(ApacheRequestExecutor.java:155)

./atlassian-bitbucket.log:2018-05-16 13:52:49,494 ERROR [Caesium-1-3]  c.a.b.i.s.c.s.p.AutomaticAuthenticationProvisioner Skipping automatic auth configuration: Elasticsearch instance is not available for connection.

./atlassian-bitbucket.log:2018-05-16 13:52:49,496 ERROR [Caesium-1-3]  c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.

Thanks,

Matt

[Michael Wolfe]

Did you ever find a resolution to this?  I too attempted to copy the user/pass into the GUI and immediately got an error.  Additionally, I tried what @Kanwar suggested and it continues to fail.

DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed due to exception java.net.ConnectException: Connection refused

Performing telnet 0 7992 also shows that nothing is running or accepting connections on that port. 

[Daniel LaFlamme]

I am having the same problem. Atlassian: could you provide some suggestions for debugging this problem? Embedded elasticsearch is not starting for me and I tried the recommendation above. The elasticsearch process is not running even though the output of the the init.d script says it is.

[Antonia V.]

You'll need to verify that elasticsearch is running. On Linux you can do this with a ps command like this:      ps -aux | grep bit

What you should see is two processes, one for elasticsearch, one for Bitbucket itself. If both are running and you still have an issue, that's when you'll need to verify the username and password that's found in your buckler.yml file.

If you don't find both processes on, the first thing to do is restart Bitbucket to see if they're now both present. If elasticsearch is still missing, we'll need to verify why elasticsearch is failing to start. This is something that would need to be done by looking through logs. 

[Daniel LaFlamme]

I verified that only the bitbucket process was running.  I then looked at /var/atlassian/application-data/bitbucket/shared/search/buckler/buckler.yml. It has the following:

auth.basic.http.enabled: true
auth.basic.tcp.enabled: true
auth.basic.username: bitbucket
auth.basic.password: bitbucket-changeit

I verified that the username and password in the Administration section of the web-ui were the same.

Restarting bitbucket, I see the following log lines related to elasticsearch:

2018-08-30 10:36:52,032 INFO  [spring-startup]  c.a.b.s.i.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob

...

2018-08-30 10:37:01,974 INFO  [Caesium-1-3]  c.a.b.s.j.c.i.HealthCheckRunner Running up to 3 search health checks
2018-08-30 10:37:02,014 INFO  [Caesium-1-3]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed due to exception java.net.ConnectException: Connection refu
sed



...

2018-08-30 10:37:26,390 ERROR [Caesium-1-3]  c.a.b.s.i.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.

The init.d script I am using to start bitbucket calls /opt/atlassian/bitbucket/5.9.1/bin/start-bitbucket.sh. It does not include the --no-search parameter.

The output of running the init.d script itself is below. Note that it says ES was started correctly.

$ sudo /etc/init.d/atlbitbucket start                                                                                      
Starting Atlassian Bitbucket as dedicated user atlbitbucket


Starting bundled Elasticsearch
        Hint: Run start-bitbucket.sh --no-search to skip starting Elasticsearch
Bundled Elasticsearch started successfully

The current open files limit is set to less than 4096
Attempting to increase limit...
        Limit increased to 4096 open files

Starting Bitbucket webapp at http://localhost:7990
The Bitbucket webapp has been started.

If you cannot access Bitbucket within 3 minutes, or encounter other issues, check the troubleshooting guide at:
https://confluence.atlassian.com/display/BitbucketServerKB/Troubleshooting+Installation
$

Are there any other logs that would be useful to check?

[Sébastien Lucchini]

Hi,

I have the same problem with Bitbucket 5.15.1 but when I start it, I can find this error on startup :

Jan 22 10:53:18 *********** bitbucket_start.sh[93422]: There was a problem starting bundled Elasticsearch

Nothing in the logs.

We have to consider prerequisites for Elasticsearch :

1/ Max file descriptors - Consider increasing to at least [65536]
In /etc/security/limits.d/, add a file [bitbucket_user].conf (soft & hard)
[bitbucket_user] soft    nofile   65536
[bitbucket_user] hard   nofile   65536

2/  Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices
In /etc/sysctl.conf  : To set this value permanently, update the vm.max_map_count   setting
             vm.max_map_count = 262144

And if you use a service, don't forget to add on it (/etc/systemd/system/bitbucket.service for example) :

LimitNOFILE=65536

After that, we can try a new Start of Bitbucket with bundled Elasticsearch and all is OK.

Best regards,

Sébastien Lucchini.