Bundled elasticsearch does not work at all for bitbucket server 5.15.0

[Leonard Li]

Bundled elasticsearch does not work at all for bitbucket server 5.15.0. Tried different suggestions but still failed.

For "Unable to synchronize the mapping in Elasticsearch" , tried: https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-connection-issues-for-elasticsearch-826900037.html?_ga=2.233302297.648682443.1542571344-1680089844.1538682649

i.e. stop-bitbucket.sh follows by start-bitbucket.sh

and then:  https://confluence.atlassian.com/bitbucketserverkb/rebuild-the-bundled-elasticsearch-indexes-for-bitbucket-server-833988382.html

i.e. curl -k -u bitbucket -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' <server_url>/rest/indexing/latest/sync

Journal below:

```

2018-11-18 17:07:07,800 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2018-11-18 17:07:07,929 INFO [I/O dispatcher 1] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401
2018-11-18 17:07:07,953 INFO [I/O dispatcher 81] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401
2018-11-18 17:07:07,958 ERROR [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch
2018-11-18 17:07:07,959 WARN [Caesium-1-3] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: xxx
2018-11-18 17:07:07,967 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2018-11-18 17:07:15,968 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Attempting provisioning in startup job
2018-11-18 17:07:15,971 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2018-11-18 17:07:15,980 ERROR [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch
2018-11-18 17:07:15,981 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: xxx
2018-11-18 17:07:31,989 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Attempting provisioning in startup job
2018-11-18 17:07:31,994 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2018-11-18 17:07:32,001 ERROR [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch
2018-11-18 17:07:32,001 WARN [Caesium-1-3] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: xxx

```

=================== log after issue "sync" =================

```

2018-11-18 17:15:14,159 WARN [analyticsEventProcessor:thread-1] bitbucket @XXX  "POST /rest/indexing/latest/sync HTTP/1.1" c.a.a.c.h.r.RemoteHashingInstructionsReader Unable to read remote instructions with key 'uid.onewayhash'.
2018-11-18 17:15:14,160 WARN [analyticsEventProcessor:thread-1] bitbucket @XXX "POST /rest/indexing/latest/sync HTTP/1.1" c.a.a.c.h.BcryptAnalyticsEmailHasher No instructions for hashing could be found.
2018-11-18 17:15:19,216 INFO [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Beginning indexing of all projects and repositories for search.
2018-11-18 17:15:19,224 ERROR [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch
2018-11-18 17:15:19,225 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob Job id: xxx

```

[Leonard Li]

The fix is to update the search password. The default does not work.

cd .../bitbucket_home/shared/search/buckler/buckler.yml

copy/paste the `auth.basic.password` to Administration|Server settings|Elasticsearch password

[koby yakov]

i had the same issue and it solved it.

thanks.

[Manish Pandya]

I am on bitbucket 5.9.0, my instance was up since 9/2018 - in 6/2019 suddenly bitbucket lost connectivity to elastic search. 

Restoring password like Leonad Li's suggestion fixed the instance.

Thanks !!

[Saurabh Raut]

Hi manish, 

I am facing same issue but my bitbucket is not even running on ip address therefore I cant do the leonad li's steps.

what should I do?

[Saurabh Raut]

I followed the same process but still test failed

[Esther Lansbergen]

Thanks, glad it solved the problem

[Ceyda Yaren]

Thanks a lot, this solved my issue!