Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to rebuild corrupted Elasticsearch index

I am using a remote Elasticsearch instance with Bitbucket. It looks like the index has become corrupt or something:

2019-03-11 10:18:09,398 ERROR [search-indexing:thread-1] c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository NSI/nsi (id: 1532) with error: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
at com.atlassian.bitbucket.internal.search.indexing.indexer.IndexStateService.lambda$save$2(IndexStateService.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)

 

Does anyone know how to rebuild the index for a remote Elasticsearch instance?

3 answers

1 accepted

1 vote
Answer accepted

You can check your index as follows:

http://<ip of elastic search instance>:9200/_cat/indices

Rebuild a corrupt index:

curl -u <admin user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' https://bitbucket.mycompany.com/rest/indexing/latest/sync

Once you do this if you check this URL again:

http://<ip of elastic search instance>:9200/_cat/indices

You should see the numbers on the right most column start to increase again:

yellow open bitbucket-project       qdfaTpJST6yQSygS-cnlyA 5 1    172 1131    52kb    52kb
yellow open bitbucket-search-v1     H0fGxz-RSB6hA_gA0lCbmw 5 1 152894 1623 715.9mb 715.9mb
yellow open bitbucket-index-state   d4fYg7BUR86kKAtJLtaPiw 5 1   1040  140 242.8kb 242.8kb
yellow open bitbucket-repository    Wu1f02P8Td6pWaBJIEDJhw 5 1   1131    0 404.5kb 404.5kb
yellow open bitbucket-index-version iedAk2sxQtW-qPgVLyoGaA 5 1      1    0   3.9kb   3.9kb

See also:

https://confluence.atlassian.com/bitbucketserverkb/elasticsearch-has-got-corrupted-index-861012994.html

Thank's for your answer,

But I have not any Elasticsearch installed (I don't know if Bitbucket Install it for me), so that i can't invoke the verification URL

http://<ip of elastic search instance>:9200/_cat/indices

Anyway, whn I try to re-index with this URL template

curl -u <admin user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' https://bitbucket.mycompany.com/rest/indexing/latest/sync

I have this error

2020-04-02 11:01:10,624 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager Attempting to schedule an ad-hoc for ElasticsearchSynchronizeJob
2020-04-02 11:01:10,625 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager There are currently 2 to select from
2020-04-02 11:01:10,662 INFO [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Beginning indexing of all projects and repositories for search.
2020-04-02 11:01:10,671 INFO [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Executing upgrade task:[Update path and filename fields for file search]
2020-04-02 11:01:16,885 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:39)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
2020-04-02 11:01:16,886 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Update path and filename fields for file search]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.
2020-04-02 11:01:16,888 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob Job id: 6e6125e9-926c-40f4-b087-f45889ed2072

I've try to stop bitbucket server, delete after backup the directory

<bitbucket-home>/shared/search/data/nodes

and restart bitbucket service, I get the same error when it try to build index...

Keep generating the underline.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

📣 Calling Bitbucket Data Center customers to participate in research

Hi everyone, Are you Bitbucket DC customer? If so, we'd love to talk to you! Our team wants to dive deep to understand your long-term plans regarding Bitbucket DC and Atlassian Cloud. Do you plan...

117 views 2 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you