Hello,
We've just upgrade Bitbucket Server from 5.16.5 to 6.3.2. Upgrade finish ok, and we have access to work but the log file shows an error:
2019-06-21 11:54:22,779 ERROR [Caesium-1-2] c.a.b.i.s.i.u.UpgradeTaskV3Point0 Failed to recreate indexjava.util.concurrent.ExecutionException: java.net.SocketTimeoutExceptionat java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)at rx.Observable.unsafeSubscribe(Observable.java:10327)at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)at rx.Observable.unsafeSubscribe(Observable.java:10327)at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)at rx.Subscriber.setProducer(Subscriber.java:211)at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)at rx.Observable.unsafeSubscribe(Observable.java:10327)at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)at rx.Subscriber.setProducer(Subscriber.java:211)at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)at rx.Observable.unsafeSubscribe(Observable.java:10327)at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)at rx.Observable.subscribe(Observable.java:10423)at rx.Observable.subscribe(Observable.java:10390)at rx.Observable.subscribe(Observable.java:10271)at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV3Point0.execute(UpgradeTaskV3Point0.java:39)at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)at java.lang.Thread.run(Thread.java:748)Caused by: java.net.SocketTimeoutException: nullat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)... 1 common frames omitted2019-06-21 11:54:22,843 ERROR [Caesium-1-2] c.a.b.i.s.i.u.UpgradeTaskV3Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''2019-06-21 11:54:22,859 ERROR [Caesium-1-2] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Create indexes for each mapping, this is required for ES 6.x support.]com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV3Point0.execute(UpgradeTaskV3Point0.java:52)at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)at java.lang.Thread.run(Thread.java:748)2019-06-21 11:54:22,860 ERROR [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.2019-06-21 11:54:22,870 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: 763a60e0-158d-44fd-b098-7d0023340bac
How could we fix it?
Thanks a lot.
Kind Regards.
Hello David,
Congratulations on making the upgrade to Bitbucket server 6.3. We do appreciate the logs you sent over and they do point to an issue with elastic search. It would seem there is an issue with creating or accessing the directory for elasticsearch. I would review the following article to attempt to resolve your issues; Rebuild the bundled Elasticsearch indexes for Bitbucket Server. Chances are you will need to review resolution #2 if your issue is with corruption. Here are the steps within that article;
(Ensure you have a good backup prior to moving forward)
Resolution #2 - delete the index If the "Resolution #1" option does not resolve the problem, perform the following steps: Stop Bitbucket Server Create a backup of <bitbucket-home>/shared/search/data/bitbucket_search directory Delete the contents of <bitbucket-home>/shared/search/data/bitbucket_search directory Start Bitbucket Server
Resolution #2 - delete the index
If the "Resolution #1" option does not resolve the problem, perform the following steps:
If you’re still having issues with elasticsearch after attempting the above please reach back out and let us know.
Regards,Stephen Sifers
Hi Stephen,
Thanks for your answer. I've already done it, but the error persist. I've also tried to remove the bitbucket_search folder and the same error were showed. After that, I've removed all the "data" folder content (which was only a folder called nodes) and now there is no errors in atlassian-bitbucket.log
Is this correct? Or bitbucket_search folder must exist, because I've made a restart of Bitbucket service and also a reboot of the machine and doesn't recreate it.
Kind Regards,
To understand fully, you removed the folders requested and then completed a reboot/service restart and the errors are no longer present. Additionally, the folder which you deleted was not recreated.
If this is the case, is Elasticsearch still not working after completing the above? Once you removed the folder and restarted the service it should have resolved the issue with Elasticsearch.
We look forward to hearing back to ensure your Elasticsearch is working properly.
Hello Stephen,
Yes, I've removed the entire folder "bitbucket_search" (and it`s content) and then I've completed a service restart (and also a system reboot) and the error persist, even so the folder "bitbucket_search" where not recreated automatically. So, after that I've removed the entire content of "data" folder (bitbucket_search folder next up level) then completed service reboot and no errors showed.
What is the real path for elasticsearch index in bitbucket?
/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/
or
/var/atlassian/application-data/bitbucket/shared/search/data/
My Elasticsearch seems to be working properly.
What's going on?
It looks like you're new here. Sign in or register to get started.
