Bitbucket Search Unavailable

[Tom Dobra]

tl;dr

Search doesn't work :-/ I've done a websearch and tried the recommended troubleshooting steps but without success. Any help would be appreciated. Thank you.

Long (sorry) version

The search function ("Search for code, commits or repositories...") stopped working a few weeks ago: "Search is currently unavailable. Refresh the page and try again" (needless to say, refreshing the page doesn't help).

I've worked through the steps on this page: https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-steps-for-bitbucket-server-code-search-827125334.html

...but I'm not having any success:

1. We're running Bitbucket Server 6.2.0 so there's no separate "atlbitbucket_search" service.

2. The Elasticsearch credentials seem OK: if I click "Test" in Server settings > Search,  Bitbucket says "Successfully connected to Elasticsearch."

3. Elasticsearch isn't hanging: it shows up in the output of ps, and entries are written to $BITBUCKET_HOME/log/search/bitbucket_search.log

4. I've enabled DEBUG level logs

5. The main log file (atlassian-bitbucket.log) contains ERRORs like:

c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository whatever/whatever (id: XXX) with error: Elasticsearch returned 503 (service unavailable) for the request

...and the search log file (bitbucket_search.log) contains WARNINGs like:

org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get[bitbucket-index-state][indexstate][1232]: routing [null]]

6. There are no search service scripts (see point 1., above)

7. The files/directories:

buckler
│ └── buckler.yml
├── elasticsearch.yml
└── logging.yml 

...are present

8.  This works:

curl -u username:password http://localhost:7992

9. This (triggering a reindex) returns nothing:

curl -u <my-admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' http://us.ourdomain:port/rest/indexing/latest/sync

10. This (checking the status of search) also returns nothing:

curl -u <my-admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' http://us.ourdomain:port/rest/indexing/latest/sync

 (...)

I've tried obvious things, like restarting the bitbucket service, (and the server as a whole).

TIA,

Tom

P.S. Ubuntu 16.04.6, Bitbucket Server 6.2.0, PostgreSQL 9.5.17

[Tom Dobra]

UPDATE

Updating to v6.6.1 doesn't resolve the issue (I'd have been surprised if it had, to be fair).

Search does actually work, if you use http://<site>:7990/plugins/servlet/search

...But the "Search" box in the banner of the page (i.e. the one with the placeholder text "Search for code, commits or repositories...") still says

Search is currently unavailable
Refresh the page and try again

[Andres LeonRangel]

thanks i will try this.

[adv4000]

Adding those 3 lines into bitbucket.properties and restarting BitBucket solved issue with Search no available.

server.secure=true
server.scheme=https
server.redirect-port=443

[Andres LeonRangel]

in our case we have a compute instance with

1. BB server
2. PostgreSQL

i saw errors regarding elastic search which I believe is an internal module of the BB application.

All I had to do to get it fixed was to reboot the server and that's it :-)

[Jeffrey Gullett]

I was having this same issue.  I was able to get the error message to go away by editing the ${BITBUCKET_HOME}\shared\search\elasticsearch.yml configuration file by changing action.auto_create_index from false to true.  Tested with Bitbucket 6.6.1/Elasticsearch 6.5.

[Tom Dobra]

Hi Jeffrey, and thanks for replying.

I've just tried this - I edited (...)shared\search\elasticsearch.yml (after backing it up), then restarted Bitbucket (it shouldn't be necessary to reboot the server) - but it hasn't made any difference: we still get the same error msg :-( 

[Jeffrey Gullett]

When you say you restarted Bitbucket, did you also restart Elasticsearch?

[Tom Dobra]

Not explicitly: we're running v6.6.1, so there's no separate service for search: starting and stopping "atlbitbucket" also starts and stops search.

[joseluisq]

I have upgraded Bitbucket Server from v5 to latest v6 and followed the steps above but without success. I get the same error again.

ERROR [spring-startup] c.a.s.s.a.l.PatchedLifecycleManager LifecycleAware.onStart() failed for component with class 'com.atlassian.bitbucket.internal.search.search.client.SearchClientLifecycleManager' from plugin 'com.atlassian.bitbucket.server.bitbucket-search'

java.lang.IllegalArgumentException: Host name may not be null

[joseluisq]

We have solved this issue.
In our case updating our ElasticSearch login credentials on Bitbucket settings (for some reason we were using the wrong ones) and then triggering a re-indexing manually again.

• If you don't know that ones you can find them at /var/atlassian/application-data/bitbucket/shared/search/buckler/buckler.yml
• For triggering a re-indexing check out https://confluence.atlassian.com/bitbucketserverkb/rebuild-the-bundled-elasticsearch-indexes-for-bitbucket-server-833988382.html

This issue can also be related to https://community.atlassian.com/t5/Bitbucket-questions/Bundled-elasticsearch-does-not-work-at-all-for-bitbucket-server/qaq-p/942796?tempId=eyJvaWRjX2NvbnNlbnRfbGFuZ3VhZ2VfdmVyc2lvbiI6IjIuMCIsIm9pZGNfY29uc2VudF9ncmFudGVkX2F0IjoxNTk1ODU4OTQwODAwfQ%3D%3D

[Tom Dobra]

Hi joseluiq, and thanks for replying. That's interesting - I'll try it again.

Right now, the main search function ("Search for code, commits or repositories...") doesn't work, but repo search ("Search for repositories...") does. We're due an upgrade; I'll keep investigating this.

[joseluisq]

Any update on this? I got the same problem for Bitbucket v5.16.10.
I have also tried to re-index with the steps above.

I always get this error:

bitbucket_app.1.z1234z12cf12@server | 2019-09-09 06:58:17,553 ERROR [spring-startup] c.a.s.s.a.l.PatchedLifecycleManager LifecycleAware.onStart() failed for component with class 'com.atlassian.bitbucket.internal.search.search.client.SearchClientLif
ecycleManager' from plugin 'com.atlassian.bitbucket.server.bitbucket-search'
bitbucket_app.1.z1234z12cf12@server | java.lang.IllegalArgumentException: Host name may not be null
bitbucket_app.1.z1234z12cf12@server | at org.apache.http.util.Args.containsNoBlanks(Args.java:81)
bitbucket_app.1.z1234z12cf12@server | at org.apache.http.HttpHost.<init>(HttpHost.java:80)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor.configure(ApacheRequestExecutor.java:102)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.elasticsearch.client.internal.InternalClient.<init>(InternalClient.java:22)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.elasticsearch.client.Client.<init>(Client.java:26)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.elasticsearch.client.Client$Builder.build(Client.java:94)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.bitbucket.internal.search.common.clientutil.DefaultElasticsearchClientFactory.createDefaultClientBuilder(DefaultElasticsearchClientFactory.java:127)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.bitbucket.internal.search.common.clientutil.DefaultElasticsearchClientFactory.create(DefaultElasticsearchClientFactory.java:65)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.bitbucket.internal.search.search.client.SearchClientLifecycleManager.refreshClient(SearchClientLifecycleManager.java:137)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.bitbucket.internal.search.search.client.SearchClientLifecycleManager.onStart(SearchClientLifecycleManager.java:124)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager$4.consume(PatchedLifecycleManager.java:311)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager$4.consume(PatchedLifecycleManager.java:308)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.notifyLifecycleAware(PatchedLifecycleManager.java:337)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.notifyOnStartIfStartedAndEnabled(PatchedLifecycleManager.java:307)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.access$300(PatchedLifecycleManager.java:34)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager$3.evaluate(PatchedLifecycleManager.java:267)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager$3.evaluate(PatchedLifecycleManager.java:264)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.notifyLifecycleAwares(PatchedLifecycleManager.java:292)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.notifyStartableLifecycleAwares(PatchedLifecycleManager.java:263)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.notifyStartableLifecycleAwaresForPlugin(PatchedLifecycleManager.java:255)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.notifyStartableLifecycleAwares(PatchedLifecycleManager.java:225)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.startIfApplicationSetup(PatchedLifecycleManager.java:208)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.sal.api.lifecycle.PatchedLifecycleManager.onPluginFrameworkStarted(PatchedLifecycleManager.java:141)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:40)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.diagnostics.internal.platform.event.EventSystemMonitor.invokeMonitored(EventSystemMonitor.java:109)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.diagnostics.internal.platform.event.MonitoredListenerInvoker.invoke(MonitoredListenerInvoker.java:38)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:38)
bitbucket_app.1.z1234z12cf12@server | at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:88)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.diagnostics.internal.platform.event.MonitoredEventDispatcher.dispatch(MonitoredEventDispatcher.java:36)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.event.internal.LockFreeEventPublisher$Publisher.dispatch(LockFreeEventPublisher.java:222)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:95)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.internalPublish(TransactionAwareEventPublisher.java:155)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.publish(TransactionAwareEventPublisher.java:99)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:83)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.plugin.manager.DefaultPluginManager.broadcastIgnoreError(DefaultPluginManager.java:2129)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.plugin.manager.DefaultPluginManager.lateStartup(DefaultPluginManager.java:678)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:528)
bitbucket_app.1.z1234z12cf12@server | at com.atlassian.stash.internal.plugin.SpringPluginSystemLifecycle.start(SpringPluginSystemLifecycle.java:35)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:350)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880)
bitbucket_app.1.z1234z12cf12@server | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
bitbucket_app.1.z1234z12cf12@server | at javax.servlet.GenericServlet.init(GenericServlet.java:158)
bitbucket_app.1.z1234z12cf12@server | at java.lang.Thread.run(Thread.java:748)

[Tom Dobra]

Hi joseluisq, and thank you for responding :)

We've spent a significant amount of time on this but without success, unfortunately. We're scheduled to upgrade soon and it's conceivable that will resolve the issue, but I'd be surprised, frankly: I haven't spotted anything in the release notes to indicate that's likely.

Sad to say, but we're starting to seriously consider moving away from Bitbucket.

I'm sorry I can't help with your problem; I hope you have more success than us. (If you do manage to solve it, feel free to share any insights you've gained - I'd be really grateful!)

Thanks again, and all the best

Tom