tl;dr
Search doesn't work :-/ I've done a websearch and tried the recommended troubleshooting steps but without success. Any help would be appreciated. Thank you.
Long (sorry) version
The search function ("Search for code, commits or repositories...") stopped working a few weeks ago: "Search is currently unavailable. Refresh the page and try again" (needless to say, refreshing the page doesn't help).
I've worked through the steps on this page: https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-steps-for-bitbucket-server-code-search-827125334.html
...but I'm not having any success:
1. We're running Bitbucket Server 6.2.0 so there's no separate "atlbitbucket_search" service.
2. The Elasticsearch credentials seem OK: if I click "Test" in Server settings > Search, Bitbucket says "Successfully connected to Elasticsearch."
3. Elasticsearch isn't hanging: it shows up in the output of ps, and entries are written to $BITBUCKET_HOME/log/search/bitbucket_search.log
4. I've enabled DEBUG level logs
5. The main log file (atlassian-bitbucket.log) contains ERRORs like:
c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository whatever/whatever (id: XXX) with error: Elasticsearch returned 503 (service unavailable) for the request
...and the search log file (bitbucket_search.log) contains WARNINGs like:
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get[bitbucket-index-state][indexstate][1232]: routing [null]]
6. There are no search service scripts (see point 1., above)
7. The files/directories:
buckler
│ └── buckler.yml
├── elasticsearch.yml
└── logging.yml
...are present
8. This works:
curl -u username:password http://localhost:7992
9. This (triggering a reindex) returns nothing:
curl -u <my-admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' http://us.ourdomain:port/rest/indexing/latest/sync
10. This (checking the status of search) also returns nothing:
curl -u <my-admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' http://us.ourdomain:port/rest/indexing/latest/sync
(...)
I've tried obvious things, like restarting the bitbucket service, (and the server as a whole).
TIA,
Tom
P.S. Ubuntu 16.04.6, Bitbucket Server 6.2.0, PostgreSQL 9.5.17
