Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Deleted user
Level
0 / 0 points
Next:
badges earned

Your Points Tracker
Challenges
Leaderboard
  • Global
  • Feed

Badge for your thoughts?

You're enrolled in our new beta rewards program. Join our group to get the inside scoop and share your feedback.

Join group
Recognition
Give the gift of kudos
You have 0 kudos available to give
Who do you want to recognize?
Why do you want to recognize them?
Kudos
Great job appreciating your peers!
Check back soon to give more kudos.

Past Kudos Given
No kudos given
You haven't given any kudos yet. Share the love above and you'll see it here.

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to rebuild corrupted Elasticsearch index

I am using a remote Elasticsearch instance with Bitbucket. It looks like the index has become corrupt or something:

2019-03-11 10:18:09,398 ERROR [search-indexing:thread-1] c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository NSI/nsi (id: 1532) with error: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
at com.atlassian.bitbucket.internal.search.indexing.indexer.IndexStateService.lambda$save$2(IndexStateService.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)

 

Does anyone know how to rebuild the index for a remote Elasticsearch instance?

1 answer

1 accepted

1 vote
Answer accepted

You can check your index as follows:

http://<ip of elastic search instance>:9200/_cat/indices

Rebuild a corrupt index:

curl -u <admin user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' https://bitbucket.mycompany.com/rest/indexing/latest/sync

Once you do this if you check this URL again:

http://<ip of elastic search instance>:9200/_cat/indices

You should see the numbers on the right most column start to increase again:

yellow open bitbucket-project       qdfaTpJST6yQSygS-cnlyA 5 1    172 1131    52kb    52kb
yellow open bitbucket-search-v1     H0fGxz-RSB6hA_gA0lCbmw 5 1 152894 1623 715.9mb 715.9mb
yellow open bitbucket-index-state   d4fYg7BUR86kKAtJLtaPiw 5 1   1040  140 242.8kb 242.8kb
yellow open bitbucket-repository    Wu1f02P8Td6pWaBJIEDJhw 5 1   1131    0 404.5kb 404.5kb
yellow open bitbucket-index-version iedAk2sxQtW-qPgVLyoGaA 5 1      1    0   3.9kb   3.9kb

See also:

https://confluence.atlassian.com/bitbucketserverkb/elasticsearch-has-got-corrupted-index-861012994.html

Thank's for your answer,

But I have not any Elasticsearch installed (I don't know if Bitbucket Install it for me), so that i can't invoke the verification URL

http://<ip of elastic search instance>:9200/_cat/indices

Anyway, whn I try to re-index with this URL template

curl -u <admin user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' https://bitbucket.mycompany.com/rest/indexing/latest/sync

I have this error

2020-04-02 11:01:10,624 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager Attempting to schedule an ad-hoc for ElasticsearchSynchronizeJob
2020-04-02 11:01:10,625 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager There are currently 2 to select from
2020-04-02 11:01:10,662 INFO [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Beginning indexing of all projects and repositories for search.
2020-04-02 11:01:10,671 INFO [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Executing upgrade task:[Update path and filename fields for file search]
2020-04-02 11:01:16,885 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:39)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
2020-04-02 11:01:16,886 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Update path and filename fields for file search]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.
2020-04-02 11:01:16,888 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob Job id: 6e6125e9-926c-40f4-b087-f45889ed2072

I've try to stop bitbucket server, delete after backup the directory

<bitbucket-home>/shared/search/data/nodes

and restart bitbucket service, I get the same error when it try to build index...

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Calling any interview participants for Bitbucket Data Center

Hi everyone,  We are looking to learn more about development teams’ workflows and pain points, especially around DevOps, integrations, administration, scale, security, and the related challeng...

528 views 6 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you