Upgrade to bitbucket 6.7 search is no longer working

[M_ Ludwig]

Hi Guys,

we recently updated to bitbucket version 6.7.

Now the data search is not working anymore. Each time we search for something we get as result:

We couldn't find any results matching
<our search string>

Try changing your query, or read the documentation about advanced search

That what we searched for got 2 results prior to the upgrade.

Can anyone please help to get our search engine working again.

Thanks a lot.

M. Ludwig

[M_ Ludwig · Accepted solution]

Hi Alexis,

thanks for reply.

Actually the atlassian filesystem is at 64% - more than 10% free space so far.

I followed your curl advice andthe visible resulton commandline was:

{"acknowledged":true}

Issued a search and voila it works.

Many many thanks for your help.

Greetings,

  Martina

[Alexis Robert]

Thanks @M_ Ludwig  ! please mark our discussion below as "Answer accepted" too so that people can easily find it. Thanks !

[Alexis Robert · Accepted solution]

Hi @M_ Ludwig ,

since you just upgraded, it might be a problem with elasticsearch (that's the component used to do the search in Bitbucket). This documentation should help you understand what's wonrg : https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-steps-for-bitbucket-server-code-search-827125334.html?_ga=2.268585526.696304845.1573393740-1867168444.1563630283

Let me know if this helps, 

--Alexis

[M_ Ludwig]

Hi Alexis,

thanks for advice.

I looked into the troubleshoot site.

Elastic is working, connection to it works perfectly.

Only thing my instance differs from advice was that a file named logging.yml is not present but a file logging.yml.bak_version2.

In the bitbucket log i find following error message concerning Elastic:

2019-11-12 00:45:04,981 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService An error was encountered while checking or creating the mapping in Elasticsearch
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: update index-version yielded 403 response code.
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultIndexVersionService.lambda$execute$6(DefaultIndexVersionService.java:107)
at io.atlassian.fugue.Either$Right.fold(Either.java:641)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultIndexVersionService.execute(DefaultIndexVersionService.java:102)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultIndexVersionService.setCurrentVersion(DefaultIndexVersionService.java:73)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgradeVersion(DefaultUpgradeService.java:60)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:52)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
2019-11-12 00:45:04,981 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: c0dbd4f5-0a28-4a07-8aa2-d0d75cc16eac

A rebuild of the search index did not solve anything. Still search does not work.

[Alexis Robert]

Hi @M_ Ludwig 

from what I've found, this could be because ElasticSearch put itself in read only mode. It is caused by the system running low on free disk space, can you check if you have enough space available on your partitions ?

If you have more than 10% free space, then it should be ok et you can try the following commands to restart ElasticSearch :

BITBUCKET_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

 Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml