Erreur d'indexation et lenteur d'accès aux dépots

[jetune@kube-cloud.com]

Bonjour,

J'ai installé une instance du serveur Bitbucket sur mes serveurs et derrière un reverse proxy NGINX.

Une fois démarré, je crée mes dépôts et j'en importe d'autres.

Je remarque par la suite que la navigation web dans mes dépôts est extrêmement lente et je me prend régulièrement des "504 Gateway timeout".

Dans les logs je me rend compte que lors du demarrage et tout au long de l'exécution du service Bitbucket, il se produit des erreurs d'indexation de chacun de mes dépôts:

2020-04-02 09:42:15,317 INFO [main] o.s.boot.SpringApplication 
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.9.RELEASE)
2020-04-02 09:42:15,872 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on vcs with PID 10254 (/opt/atlassian/bitbucket/app/WEB-INF/classes started by bitbucket in /)
2020-04-02 09:42:15,874 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2020-04-02 09:42:17,799 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 6.10.1 (5e5d5d4 built on Mon Feb 10 22:47:35 UTC 2020)
2020-04-02 09:42:17,799 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation OpenJDK 64-Bit Server VM 11.0.1+13
2020-04-02 09:42:20,360 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 10.582 seconds (JVM running for 11.695)
2020-04-02 09:42:26,098 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /kis/vcs/datas
2020-04-02 09:43:31,838 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /kis/vcs/datas/shared
2020-04-02 09:43:36,911 WARN [spring-startup] com.hazelcast.instance.Node [172.16.1.62]:5701 [bitbucket] [3.11.1] No join method is enabled! Starting standalone.
2020-04-02 09:44:29,312 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:33501
2020-04-02 09:44:31,698 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:41795
2020-04-02 09:44:39,219 WARN [FelixStartLevel] o.e.g.b.e.i.s.ExtenderConfiguration Gemini Blueprint extensions bundle not present, annotation processing disabled.
2020-04-02 09:46:21,108 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.16.1.62]:5701 [bitbucket] [3.11.1] BroadcastOperationControlTask delayed 14248 ms
2020-04-02 09:47:13,976 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/plugin/connect/confluence/ in bundle com.atlassian.plugins.atlassian-connect-plugin; it does not exist
2020-04-02 09:47:14,100 WARN [spring-startup] c.a.p.c.r.StaticPluginPathScanner Cannot scan directory /static/bitbucket/internal/feature/admin/ratelimit/ in web static as it does not exist
2020-04-02 09:47:16,034 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2020-04-02 09:47:16,582 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2020-04-02 09:47:16,594 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2020-04-02 09:47:16,602 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2020-04-02 09:47:16,986 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2020-04-02 09:47:17,209 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2020-04-02 09:47:17,750 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2020-04-02 09:47:17,760 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2020-04-02 09:47:17,791 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2020-04-02 09:47:17,820 INFO [spring-startup] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2020-04-02 09:47:17,820 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2020-04-02 09:47:17,821 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2020-04-02 09:47:18,176 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/bin/git version 2.17.1
2020-04-02 09:47:33,022 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2020-04-02 09:47:33,781 ERROR [Caesium-1-3] c.a.b.i.s.i.s.StreamingComparator Unable to perform job due to an unexpected error
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: The request to Elasticsearch failed with HTTP status code: 503
at com.atlassian.bitbucket.internal.search.indexing.util.ElderScroll$ScrollProducer.lambda$fetchMore$2(ElderScroll.java:134)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.internal.producers.ProducerArbiter.setProducer(ProducerArbiter.java:126)
at rx.internal.operators.OnSubscribeSwitchIfEmpty$ParentSubscriber.setProducer(OnSubscribeSwitchIfEmpty.java:76)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeSwitchIfEmpty$ParentSubscriber.subscribe(OnSubscribeSwitchIfEmpty.java:104)
at rx.internal.operators.OnSubscribeSwitchIfEmpty.call(OnSubscribeSwitchIfEmpty.java:52)
at rx.internal.operators.OnSubscribeSwitchIfEmpty.call(OnSubscribeSwitchIfEmpty.java:31)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.util.ElderScroll$ScrollProducer.fetchMore(ElderScroll.java:157)
at com.atlassian.bitbucket.internal.search.indexing.util.ElderScroll$ScrollProducer.call(ElderScroll.java:84)
at com.atlassian.bitbucket.internal.search.indexing.util.ElderScroll.lambda$search$0(ElderScroll.java:56)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
at rx.internal.operators.OnSubscribeReduceSeed.call(OnSubscribeReduceSeed.java:40)
at rx.internal.operators.OnSubscribeReduceSeed.call(OnSubscribeReduceSeed.java:24)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:64)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consumeSingle(Observables.java:92)
at com.atlassian.bitbucket.internal.search.indexing.syncutil.StreamingComparator.doComparison(StreamingComparator.java:95)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.compareObservables(IndexingSynchronizationService.java:254)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.compareEntityStreams(IndexingSynchronizationService.java:236)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.lambda$synchronizeRepositories$3(IndexingSynchronizationService.java:183)
at com.atlassian.stash.internal.user.DefaultEscalatedSecurityContext.call(DefaultEscalatedSecurityContext.java:58)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeRepositories(IndexingSynchronizationService.java:178)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:94)
at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: com.atlassian.elasticsearch.client.search.SearchResponse.class
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
... 68 common frames omitted
2020-04-02 09:47:33,793 WARN [Caesium-1-3] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: caebc81e-99b0-49d8-8160-0dd6890e1937
2020-04-02 09:47:33,887 ERROR [search-indexing:thread-1] c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for project UTL (id: 3) with error: Index response returned status code 500. Response: IndexResponse{statusCode=500, content={
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[bitbucket_bundled][127.0.0.1:7993][indices:data/write/bulk[s][p]]"
}
],
"type": "file_system_exception",
"reason": "/kis/vcs/datas/shared/search/data/nodes/0/indices/ArqSjN-DQ8-UyroAypD9yA/4/index/_d.fdt: Too many open files"
},
"status": 500
}}
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 500. Response: IndexResponse{statusCode=500, content={
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[bitbucket_bundled][127.0.0.1:7993][indices:data/write/bulk[s][p]]"
}
],
"type": "file_system_exception",
"reason": "/kis/vcs/datas/shared/search/data/nodes/0/indices/ArqSjN-DQ8-UyroAypD9yA/4/index/_d.fdt: Too many open files"
},
"status": 500
}}

at com.atlassian.bitbucket.internal.search.indexing.indexer.IndexRequestDispatcher.convertIndexResponse(IndexRequestDispatcher.java:252)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:235)
at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10230)
at 

Mes soupcons de lenteur s'orientent vers cette erreur d'indexation des dépôts qui je pensent est sencé améliorer les accès.

Par contre je n'en comprends pas la raison ni la source.

Il y aurait-il un moyen de résoudre ce problème ?