Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Resetting heartbeat timestamps because of huge system clock jump

We have a fresh Bitbucket Server installed in a VM containing Jira/Confluence/Bitbucket/Jenkins apps.

Installation was made with docker-compose files and behind Traefik reverse proxy.

All is working fine except Bitbucket. Sometine pages take a while to load (few minutes) but finaly load and logs display Hazelcast clocl jump errors and monitor delayd errors.

Starting Bitbucket webapp at http://localhost:7990
2021-01-04 22:25:55,593 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.3.RELEASE)


2021-01-04 22:25:55,696 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v7.8.1 on a089f474d6c3 with PID 131 (/opt/atlassian/bitbucket/app/WEB-INF/classes started by bitbucket in /var/atlassian/application-data/bitbucket)
2021-01-04 22:25:55,697 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2021-01-04 22:25:56,290 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 7.8.1 (6433dff built on Fri Dec 11 00:38:29 CET 2020)
2021-01-04 22:25:56,290 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: AdoptOpenJDK OpenJDK 64-Bit Server VM 1.8.0_275-b01
2021-01-04 22:25:57,272 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 2.688 seconds (JVM running for 3.132)
2021-01-04 22:25:59,594 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2021-01-04 22:26:02,354 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2021-01-04 22:26:03,690 WARN [spring-startup] com.hazelcast.instance.Node [172.18.0.4]:5701 [bitbucket] [3.12.9] No join method is enabled! Starting standalone.
2021-01-04 22:26:13,127 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:37109
2021-01-04 22:26:15,575 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:38509
2021-01-04 22:26:17,025 INFO [spring-startup] c.a.s.i.s.g.h.DefaultGitHookService Git hook callback socket listening on 127.0.0.1:39607
2021-01-04 22:26:18,939 WARN [FelixStartLevel] o.e.g.b.e.i.s.ExtenderConfiguration Gemini Blueprint extensions bundle not present, annotation processing disabled.
2021-01-04 22:26:28,220 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle com.atlassian.bitbucket.server.bitbucket-frontend; it does not exist
2021-01-04 22:26:28,558 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/plugin/connect/confluence/ in bundle com.atlassian.plugins.atlassian-connect-plugin; it does not exist
2021-01-04 22:26:28,604 WARN [spring-startup] c.a.p.c.r.OsgiBundledPathScanner Cannot scan directory /page/ in bundle com.atlassian.bitbucket.server.bitbucket-policies as it does not exist
2021-01-04 22:26:28,616 WARN [spring-startup] c.a.p.c.r.StaticPluginPathScanner Cannot scan directory /static/bitbucket/internal/feature/admin/ratelimit/ in web static as it does not exist
2021-01-04 22:26:29,224 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2021-01-04 22:26:29,370 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2021-01-04 22:26:29,371 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2021-01-04 22:26:29,373 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2021-01-04 22:26:29,374 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2021-01-04 22:26:29,376 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2021-01-04 22:26:29,377 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2021-01-04 22:26:29,561 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2021-01-04 22:26:29,676 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2021-01-04 22:26:29,835 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2021-01-04 22:26:29,838 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2021-01-04 22:26:29,861 INFO [spring-startup] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2021-01-04 22:26:29,861 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: VersionCheck
2021-01-04 22:26:30,100 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/bin/git version 2.29.2
2021-01-04 22:26:44,853 INFO [Caesium-1-4] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2021-01-04 22:33:52,608 WARN [hook-callback:thread-1] devops @B25WX0x1353x85x0 192.168.1.21,172.18.0.3 "POST /scm/devops/leicapps-dockers.git/git-receive-pack HTTP/1.1" org.eclipse.jgit.util.FS Exception caught during execution of command '[git, config, --system, --edit]' in '/usr/bin', return code '128', error message 'fatal: Invalid path '/usr/etc': No such file or directory
'
2021-01-05 00:33:57,786 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6686087 ms
2021-01-05 00:33:59,771 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6686087 ms, Heartbeat-Timeout: 60000 ms
2021-01-04 22:43:03,686 WARN [hz.hazelcast.cached.thread-6] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6686084 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:09:57,746 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685993 ms
2021-01-05 01:09:59,734 WARN [hz.hazelcast.cached.thread-6] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685993 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:09:59,747 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] BroadcastOperationControlTask delayed 6685996 ms
2021-01-04 23:19:03,745 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685989 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 03:02:57,633 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685695 ms
2021-01-05 03:02:59,621 WARN [hz.hazelcast.cached.thread-2] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685695 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:12:08,929 WARN [hz.hazelcast.cached.thread-2] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685692 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 03:48:57,587 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685575 ms
2021-01-05 03:48:59,575 WARN [hz.hazelcast.cached.thread-8] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685575 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 01:58:09,003 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685572 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 04:03:57,573 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685535 ms
2021-01-05 04:04:57,572 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685532 ms
2021-01-05 04:05:57,571 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685530 ms
2021-01-05 04:06:57,571 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685528 ms
2021-01-05 04:07:57,570 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685525 ms
2021-01-05 04:08:57,569 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685522 ms
2021-01-05 04:09:57,567 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685519 ms
2021-01-05 04:10:57,566 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685517 ms
2021-01-05 04:11:57,565 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685514 ms
2021-01-05 04:12:57,564 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685512 ms
2021-01-05 04:13:57,563 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685509 ms
2021-01-05 04:14:57,561 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685506 ms
2021-01-05 04:15:57,560 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685503 ms
2021-01-05 04:15:59,549 WARN [hz.hazelcast.cached.thread-6] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685503 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 02:25:09,048 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685500 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 04:16:57,561 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685501 ms
2021-01-05 04:17:57,560 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685499 ms
2021-01-05 04:18:57,559 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685496 ms
2021-01-05 04:19:57,559 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685494 ms
2021-01-05 04:20:57,558 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685492 ms
2021-01-05 04:21:57,556 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685488 ms
2021-01-05 04:47:58,529 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685420 ms
2021-01-05 04:47:59,517 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685420 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 02:57:09,100 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685417 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 05:39:58,476 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6685284 ms
2021-01-05 05:39:59,464 WARN [hz.hazelcast.cached.thread-3] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6685284 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 05:39:59,477 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] BroadcastOperationControlTask delayed 6685287 ms
2021-01-05 03:49:09,184 WARN [hz.hazelcast.cached.thread-1] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6685281 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 07:30:58,361 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6684993 ms
2021-01-05 07:30:59,349 WARN [hz.hazelcast.cached.thread-4] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6684992 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 05:40:09,359 WARN [hz.hazelcast.cached.thread-8] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6684989 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 08:09:58,322 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6684891 ms
2021-01-05 08:09:59,310 WARN [hz.hazelcast.cached.thread-4] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6684891 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 08:09:59,323 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] BroadcastOperationControlTask delayed 6684894 ms
2021-01-05 06:19:09,422 WARN [hz.hazelcast.cached.thread-8] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6684888 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 10:03:58,203 WARN [hz.hazelcast.InvocationMonitorThread] c.h.s.i.o.impl.InvocationMonitor [172.18.0.4]:5701 [bitbucket] [3.12.9] MonitorInvocationsTask delayed 6684592 ms
2021-01-05 10:03:59,191 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 6684592 ms, Heartbeat-Timeout: 60000 ms
2021-01-05 08:13:09,601 WARN [hz.hazelcast.cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager [172.18.0.4]:5701 [bitbucket] [3.12.9] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: -6684589 ms, Heartbeat-Timeout: 60000 ms

Hosting VM is Unbuntu 20.04, 32Go RAM, HDD disk and 8 cores cpu 2.29GHz

1 answer

1 accepted

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

3,752 views 10 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you