Missed Team ’24? Catch up on announcements here.

×
Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Products
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

Elasticsearche error after upgrade

David Patiño
David Patiño June 21, 2019

Hello,

We've just upgrade Bitbucket Server from 5.16.5 to 6.3.2. Upgrade finish ok, and we have access to work but the log file shows an error:

 

2019-06-21 11:54:22,779 ERROR [Caesium-1-2] c.a.b.i.s.i.u.UpgradeTaskV3Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV3Point0.execute(UpgradeTaskV3Point0.java:39)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
2019-06-21 11:54:22,843 ERROR [Caesium-1-2] c.a.b.i.s.i.u.UpgradeTaskV3Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''
2019-06-21 11:54:22,859 ERROR [Caesium-1-2] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Create indexes for each mapping, this is required for ES 6.x support.]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV3Point0.execute(UpgradeTaskV3Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
2019-06-21 11:54:22,860 ERROR [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.
2019-06-21 11:54:22,870 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: 763a60e0-158d-44fd-b098-7d0023340bac

 

How could we fix it?

 

Thanks a lot.

Kind Regards.

Answer
Watch
Like Stephen Sifers likes this
1036 views

2 answers

0 votes
Danzel Washington
Danzel Washington
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
Get involved
October 11, 2019

What's going on?

Reply
0 votes
Stephen Sifers
Stephen Sifers
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 24, 2019

Hello David,

Congratulations on making the upgrade to Bitbucket server 6.3. We do appreciate the logs you sent over and they do point to an issue with elastic search. It would seem there is an issue with creating or accessing the directory for elasticsearch. I would review the following article to attempt to resolve your issues; Rebuild the bundled Elasticsearch indexes for Bitbucket Server. Chances are you will need to review resolution #2 if your issue is with corruption. Here are the steps within that article;

(Ensure you have a good backup prior to moving forward)

Resolution #2 - delete the index

If the "Resolution #1" option does not resolve the problem, perform the following steps:

  1. Stop Bitbucket Server
  2. Create a backup of <bitbucket-home>/shared/search/data/bitbucket_search directory
  3. Delete the contents of <bitbucket-home>/shared/search/data/bitbucket_search directory
  4. Start Bitbucket Server

If you’re still having issues with elasticsearch after attempting the above please reach back out and let us know.

Regards,
Stephen Sifers

View More Comments
David Patiño
David Patiño July 2, 2019

Hi Stephen,

Thanks for your answer. I've already done it, but the error persist. I've also tried to remove the bitbucket_search folder and the same error were showed. After that, I've removed all the "data" folder content (which was only a folder called nodes) and now there is no errors in atlassian-bitbucket.log

Is this correct? Or bitbucket_search folder must exist, because I've made a restart of Bitbucket service and also a reboot of the machine and doesn't recreate it.

Thanks a lot.

Kind Regards,

Like
Stephen Sifers
Stephen Sifers
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 5, 2019

Hello David,

To understand fully, you removed the folders requested and then completed a reboot/service restart and the errors are no longer present. Additionally, the folder which you deleted was not recreated.

If this is the case, is Elasticsearch still not working after completing the above? Once you removed the folder and restarted the service it should have resolved the issue with Elasticsearch.

We look forward to hearing back to ensure your Elasticsearch is working properly.

Regards,
Stephen Sifers

Like
David Patiño
David Patiño July 8, 2019

Hello Stephen,

Yes, I've removed the entire folder "bitbucket_search" (and it`s content) and then I've completed a service restart (and also a system reboot) and the error persist, even so the folder "bitbucket_search" where not recreated automatically. So, after that I've removed the entire content of "data" folder (bitbucket_search folder next up level) then completed service reboot and no errors showed.

What is the real path for elasticsearch index in bitbucket?

/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/

or

/var/atlassian/application-data/bitbucket/shared/search/data/

My Elasticsearch seems to be working properly.

Thanks a lot.

Kind Regards,

Like
Stephen Sifers
Stephen Sifers
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 8, 2019

Hello David,

Glad to hear Elasticsearch is working again.

The directories needed for Elasticsearch are as follows:

  • LOG_DIR
    • %BITBUCKET_HOME%\log\search
  • DATA_DIR
    • %BITBUCKET_HOME%\shared\search\data
  • CONF_DIR
    • %BITBUCKET_HOME%\shared\search

Source documentation: How to change data location for Elasticsearch.

Regards,
Stephen Sifers

Like
David Patiño
David Patiño July 11, 2019

Hello Stephen,

OK, thanks a lot.

Kind Regards,

Like • Stephen Sifers likes this
Reply

Suggest an answer

Log in or Sign up to answer
Bitbucket
Bitbucket
TAGS
AUG Leaders

Atlassian Community Events

    See all events