I am using a remote Elasticsearch instance with Bitbucket. It looks like the index has become corrupt or something:
2019-03-11 10:18:09,398 ERROR [search-indexing:thread-1] c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository NSI/nsi (id: 1532) with error: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
at com.atlassian.bitbucket.internal.search.indexing.indexer.IndexStateService.lambda$save$2(IndexStateService.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
Does anyone know how to rebuild the index for a remote Elasticsearch instance?
You can check your index as follows:
http://<ip of elastic search instance>:9200/_cat/indices
Rebuild a corrupt index:
curl -u <admin user> -X POST -H
'Content-Type: application/json'
-H
'Accept: application/json'
https:
//bitbucket.mycompany.com/rest/indexing/latest/sync
Once you do this if you check this URL again:
http://<ip of elastic search instance>:9200/_cat/indices
You should see the numbers on the right most column start to increase again:
yellow open bitbucket-project qdfaTpJST6yQSygS-cnlyA 5 1 172 1131 52kb 52kb yellow open bitbucket-search-v1 H0fGxz-RSB6hA_gA0lCbmw 5 1 152894 1623 715.9mb 715.9mb yellow open bitbucket-index-state d4fYg7BUR86kKAtJLtaPiw 5 1 1040 140 242.8kb 242.8kb yellow open bitbucket-repository Wu1f02P8Td6pWaBJIEDJhw 5 1 1131 0 404.5kb 404.5kb yellow open bitbucket-index-version iedAk2sxQtW-qPgVLyoGaA 5 1 1 0 3.9kb 3.9kb
See also:
Thank's for your answer,
But I have not any Elasticsearch installed (I don't know if Bitbucket Install it for me), so that i can't invoke the verification URL
http://<ip of elastic search instance>:9200/_cat/indices
Anyway, whn I try to re-index with this URL template
curl -u <admin user> -X POST -H
'Content-Type: application/json'
-H
'Accept: application/json'
https:
//bitbucket.mycompany.com/rest/indexing/latest/sync
I have this error
2020-04-02 11:01:10,624 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager Attempting to schedule an ad-hoc for ElasticsearchSynchronizeJob
2020-04-02 11:01:10,625 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager There are currently 2 to select from
2020-04-02 11:01:10,662 INFO [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Beginning indexing of all projects and repositories for search.
2020-04-02 11:01:10,671 INFO [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Executing upgrade task:[Update path and filename fields for file search]
2020-04-02 11:01:16,885 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:39)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
2020-04-02 11:01:16,886 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Update path and filename fields for file search]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.
2020-04-02 11:01:16,888 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob Job id: 6e6125e9-926c-40f4-b087-f45889ed2072
I've try to stop bitbucket server, delete after backup the directory
<bitbucket-home>/shared/search/data/nodes
and restart bitbucket service, I get the same error when it try to build index...
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.