Resetting heartbeat timestamps because of huge system clock jump

[Rémy]

We have a fresh Bitbucket Server installed in a VM containing Jira/Confluence/Bitbucket/Jenkins apps.

Installation was made with docker-compose files and behind Traefik reverse proxy.

All is working fine except Bitbucket. Sometine pages take a while to load (few minutes) but finaly load and logs display Hazelcast clocl jump errors and monitor delayd errors.

Starting Bitbucket webapp at http://localhost:7990
2021-01-04 22:25:55,593 INFO [main] o.s.boot.SpringApplication 
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.3.RELEASE)


2021-01-04 22:25:55,696 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v7.8.1 on a089f474d6c3 with PID 131 (/opt/atlassian/bitbucket/app/WEB-INF/classes started by bitbucket in /var/atlassian/application-data/bitbucket)
2021-01-04 22:25:55,697 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2021-01-04 22:25:56,290 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 7.8.1 (6433dff built on Fri Dec 11 00:38:29 CET 2020)
2021-01-04 22:25:56,290 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: AdoptOpenJDK OpenJDK 64-Bit Server VM 1.8.0_275-b01
2021-01-04 22:25:57,272 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 2.688 seconds (JVM running for 3.132)
2021-01-04 22:25:59,594 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2021-01-04 22:26:02,354 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2021-01-04 22:26:03,690 WARN [spring-startup] com.hazelcast.instance.Node [172.18.0.4]:5701 [bitbucket] [3.12.9] No join method is enabled! Starting standalone.
2021-01-04 22:26:13,127 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:37109
2021-01-04 22:26:15,575 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:38509
2021-01-04 22:26:17,025 INFO [spring-startup] c.a.s.i.s.g.h.DefaultGitHookService Git hook callback socket listening on 127.0.0.1:39607
2021-01-04 22:26:18,939 WARN [FelixStartLevel] o.e.g.b.e.i.s.ExtenderConfiguration Gemini Blueprint extensions bundle not present, annotation processing disabled.
2021-01-04 22:26:28,220 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle com.atlassian.bitbucket.server.bitbucket-frontend; it does not exist
2021-01-04 22:26:28,558 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/plugin/connect/confluence/ in bundle com.atlassian.plugins.atlassian-connect-plugin; it does not exist
2021-01-04 22:26:28,604 WARN [spring-startup] c.a.p.c.r.OsgiBundledPathScanner Cannot scan directory /page/ in bundle com.atlassian.bitbucket.server.bitbucket-policies as it does not exist
2021-01-04 22:26:28,616 WARN [spring-startup] c.a.p.c.r.StaticPluginPathScanner Cannot scan directory /static/bitbucket/internal/feature/admin/ratelimit/ in web static as it does not exist
2021-01-04 22:26:29,224 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2021-01-04 22:26:29,370 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2021-01-04 22:26:29,371 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2021-01-04 22:26:29,373 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2021-01-04 22:26:29,374 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2021-01-04 22:26:29,376 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2021-01-04 22:26:29,377 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2021-01-04 22:26:29,561 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2021-01-04 22:26:29,676 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2021-01-04 22:26:29,835 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2021-01-04 22:26:29,861 INFO [spring-startup] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2021-01-04 22:26:29,861 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: VersionCheck
2021-01-04 22:26:30,100 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/bin/git version 2.29.2
2021-01-04 22:26:44,853 INFO [Caesium-1-4] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2021-01-04 22:33:52,608 WARN [hook-callback:thread-1] devops @B25WX0x1353x85x0 192.168.1.21,172.18.0.3 "POST /scm/devops/leicapps-dockers.git/git-receive-pack HTTP/1.1" org.eclipse.jgit.util.FS Exception caught during execution of command '[git, config, --system, --edit]' in '/usr/bin', return code '128', error message 'fatal: Invalid path '/usr/etc': No such file or directory
'
2021-01-05 00:33:57,786 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6686087 ms
2021-01-05 00:33:59,771 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6686087 ms, Heartbeat-Timeout: 60000 ms
2021-01-04 22:43:03,686 WARN [hz.hazelcast.cached.thread-6] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6686084 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:09:57,746 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685993 ms
2021-01-05 01:09:59,734 WARN [hz.hazelcast.cached.thread-6] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685993 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:09:59,747 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] BroadcastOperationControlTask delayed 6685996 ms
2021-01-04 23:19:03,745 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685989 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 03:02:57,633 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685695 ms
2021-01-05 03:02:59,621 WARN [hz.hazelcast.cached.thread-2] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685695 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:12:08,929 WARN [hz.hazelcast.cached.thread-2] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685692 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 03:48:57,587 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685575 ms
2021-01-05 03:48:59,575 WARN [hz.hazelcast.cached.thread-8] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685575 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:58:09,003 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685572 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 04:03:57,573 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685535 ms
2021-01-05 04:04:57,572 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685532 ms
2021-01-05 04:05:57,571 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685530 ms
2021-01-05 04:06:57,571 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685528 ms
2021-01-05 04:07:57,570 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685525 ms
2021-01-05 04:08:57,569 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685522 ms
2021-01-05 04:09:57,567 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685519 ms
2021-01-05 04:10:57,566 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685517 ms
2021-01-05 04:11:57,565 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685514 ms
2021-01-05 04:12:57,564 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685512 ms
2021-01-05 04:13:57,563 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685509 ms
2021-01-05 04:14:57,561 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685506 ms
2021-01-05 04:15:57,560 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685503 ms
2021-01-05 04:15:59,549 WARN [hz.hazelcast.cached.thread-6] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685503 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 02:25:09,048 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685500 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 04:16:57,561 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685501 ms
2021-01-05 04:17:57,560 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685499 ms
2021-01-05 04:18:57,559 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685496 ms
2021-01-05 04:19:57,559 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685494 ms
2021-01-05 04:20:57,558 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685492 ms
2021-01-05 04:21:57,556 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685488 ms
2021-01-05 04:47:58,529 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685420 ms
2021-01-05 04:47:59,517 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685420 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 02:57:09,100 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685417 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 05:39:58,476 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685284 ms
2021-01-05 05:39:59,464 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685284 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 05:39:59,477 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] BroadcastOperationControlTask delayed 6685287 ms
2021-01-05 03:49:09,184 WARN [hz.hazelcast.cached.thread-1] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685281 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 07:30:58,361 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6684993 ms
2021-01-05 07:30:59,349 WARN [hz.hazelcast.cached.thread-4] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6684992 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 05:40:09,359 WARN [hz.hazelcast.cached.thread-8] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6684989 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 08:09:58,322 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6684891 ms
2021-01-05 08:09:59,310 WARN [hz.hazelcast.cached.thread-4] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6684891 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 08:09:59,323 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] BroadcastOperationControlTask delayed 6684894 ms
2021-01-05 06:19:09,422 WARN [hz.hazelcast.cached.thread-8] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6684888 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 10:03:58,203 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6684592 ms
2021-01-05 10:03:59,191 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6684592 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 08:13:09,601 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6684589 ms, Heartbeat-Timeout: 60000 ms

Hosting VM is Unbuntu 20.04, 32Go RAM, HDD disk and 8 cores cpu 2.29GHz

[Gonchik Tsymzhitov · Accepted solution]

it's like oom error. Because out of memory in your situation made the problem

https://github.com/hazelcast/hazelcast/issues/8827

https://stackoverflow.com/questions/41838025/hazelcast-warning-resetting-heartbeat-timestamps-because-of-huge-system-clock

[E. S. Souvenir SAMA]

I's ok