We are running bitbucket v 7.0.1 and now getting the following issue where the search seems to be hung up (or has stopped) :
2022-11-14 01:37:22,322 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
I then turned on the debugging and restarted the bitbucket and see where where the logs keep growing with rows looking like the below :
tingJobRunner startedAt=Mon Nov 14 08:06:47 EST 2022 runOutcome=SUCCESS message=Successfully update throttle limits
2022-11-14 08:06:50,504 DEBUG [Catalina-utility-2] o.a.catalina.session.ManagerBase Start expire sessions StandardManager at 1668431210504 sessioncount 0
2022-11-14 08:06:50,504 DEBUG [Catalina-utility-2] o.a.catalina.session.ManagerBase End expire sessions StandardManager processingTime 0 expired sessions: 0
2022-11-14 08:06:51,446 DEBUG [hz.hazelcast.cached.thread-1] c.h.i.p.InternalPartitionService [207.254.3.93]:5701 [atlbitbucket] [3.12.6] Checking partition state, version: 272
2022-11-14 08:06:52,290 DEBUG [java-sdk-http-connection-reaper] o.a.h.i.c.PoolingHttpClientConnectionManager Closing connections idle longer than 60000 MILLISECONDS
2022-11-14 08:06:52,346 DEBUG [Caesium-1-3] c.a.s.c.impl.SchedulerQueueImpl take: null (loop)
2022-11-14 08:06:52,346 DEBUG [Caesium-1-3] c.a.s.c.impl.SchedulerQueueImpl take: QueuedJob[jobId=ThrottleAdjustingJobRunner,deadline=1668431212346]
2022-11-14 08:06:52,346 DEBUG [Caesium-1-2] c.a.s.c.impl.SchedulerQueueImpl take: null (loop)
2022-11-14 08:06:52,346 DEBUG [Caesium-1-4] c.a.s.c.impl.SchedulerQueueImpl take: null (loop)
2022-11-14 08:06:52,346 DEBUG [Caesium-1-3] c.a.s.c.impl.SchedulerQueueImpl add job=QueuedJob[jobId=ThrottleAdjustingJobRunner,deadline=1668431217346] replaced=null
2022-11-14 08:06:52,346 DEBUG [Caesium-1-1] c.a.s.c.impl.SchedulerQueueImpl take: null (loop)
2022-11-14 08:06:52,346 DEBUG [Caesium-1-2] c.a.s.c.impl.SchedulerQueueImpl take: null (loop)
2022-11-14 08:06:52,346 DEBUG [Caesium-1-3] c.a.s.c.impl.CaesiumSchedulerService Enqueued job 'ThrottleAdjustingJobRunner' for Mon Nov 14 08:06:57 EST 2022
2022-11-14 08:06:52,346 DEBUG [Caesium-1-3] c.a.scheduler.core.JobLauncher launch: RUN_LOCALLY: ThrottleAdjustingJobRunner
2022-11-14 08:06:52,347 DEBUG [Caesium-1-3] c.a.s.core.AbstractSchedulerService addRunDetails: jobId=ThrottleAdjustingJobRunner startedAt=Mon Nov 14 08:06:52 EST 2022 runOutcome=SUCCESS message=Successfully update throttle limits
I am not sure what exactly the issue is , but some assistance would be greatly appreciated.
Maybe useful to know :
1. We were initially getting an error in the logs which we solved by following the steps here:
https://confluence.atlassian.com/bitbucketserverkb/elasticsearch-reports-http-status-403-in-bitbucket-logs-968656362.html
The reported issue above happened after the original problem was resolved
Regards,
Richard.
