Further ideas on 401 errors with elasticserarch

Question

This question is in reference to Atlassian Documentation: Troubleshooting Connection Issues for Elasticsearch

Although this appears to be the issue we are having, the solution does not resolve our issue.

We followed the resolution outlined above but are still getting 401 errors and the username and password fields are empty in the admin UI, although the elasticserarch home URL is there.

There are no relevant entries in the bitbucket.properties file.

As part of the attempt to ensure things were starting correctly I changed from SysV init scripts to systemd.  systemctl status bitbucket shows both processes started.

Our environment is Ubuntu 16.04LTS and Bitbucket 4.8.2

Michael Lightfo · Answer

Thanks Kristy.

I was not altogether accurate.  We upgraded first using a test database, then cut over to our production database which is when the problem occurred.

I will wait for Atlassian to look at our ticket.

Kristy · Answer

Hi Michael,

What version did you upgrade from? If it was earlier than 4.6 then the elasticsearch instance would not have been running, so the upgrade wouldn't have "broken" it in that sense. Another thing that might impact it is whether you are running a datacenter instance of Bitbucket or not.

The connection tester is coming back with a 401 which means that elasticsearch is set up with a password that Bitbucket does not know about.

I really recommend waiting for support's response to this, since they have access to all of the logs and details about your Bitbucket Home and Elasticsearch Home, and whether you are running with Data Center or not.

However, details on where to find the Elasticsearch password (configured by the Buckler plugin in Elasticsearch) can be found here: https://confluence.atlassian.com/display/BitbucketServer/Install+and+configure+a+remote+Elasticsearch+instance

Michael Lightfo · Answer

We've done that and are waiting for a response.

I've also done a great deal of ferreting around in the bowels of Bitbucket and elasticsearch.

I can report that the problem is related to Bitbucket not connecting to the search:

2016-09-01 08:02:35,702 INFO  [I/O dispatcher 9] lig047 @S1KM40x482x71478x0 1vyzj3z 152.83.229.24,0:0:0:0:0:0:0:1 "POST /rest/search-admin/latest/test-connection HTTP/1.1" c.a.b.s.s.i.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401

2016-09-01 08:03:07,592 INFO  [I/O dispatcher 13] lig047 @S1KM40x483x71568x0 1vyzj3z 152.83.229.24,0:0:0:0:0:0:0:1 "POST /rest/search-admin/latest/test-connection HTTP/1.1" c.a.b.s.s.i.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401

2016-09-01 08:05:00,034 INFO  [Caesium-1-1]  c.a.b.s.i.i.jobs.StartupChecksJob Attempting provisioning in startup job

2016-09-01 08:05:00,037 INFO  [Caesium-1-1]  c.a.b.s.i.i.jobs.StartupChecksJob Running startup jobs for search

2016-09-01 08:05:00,041 ERROR [Caesium-1-1]  c.a.b.s.i.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch

2016-09-01 08:05:00,041 WARN  [Caesium-1-1]  c.a.b.s.i.i.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.search.internal.indexing.jobs.StartupChecksJob  Job id: 91e10c10-712b-4486-b768-6f0592a24a4c

On our test system it all works.  Something in the upgrade process on production seems to have broken it.