Bitbucket taking 35 minutes to start

[BlueLetterBible]

We came in this morning to a non-responsive Bibucket server.  Restarting, it is taking over 30 minutes to complete a start up and become available.

2019-02-11 16:37:14,896 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.17.RELEASE)


2019-02-11 16:37:22,268 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on sc-livemedia2.sowingcircle.net with PID 7133 (/opt/atlassian/bitbucket/5.16.0/app/WEB-INF/classes started by atlbitbucket in /)
2019-02-11 16:37:22,278 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2019-02-11 16:37:59,232 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.16.0 (c03f0a7 built on Thu Nov 15 20:10:50 PST 2018)
2019-02-11 16:37:59,236 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_191-b12
2019-02-11 16:39:01,017 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 135.944 seconds (JVM running for 217.121)
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-2] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [208] milliseconds.
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-8] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [210] milliseconds.
2019-02-11 16:39:40,775 WARN [http-nio-7990-exec-7] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [209] milliseconds.
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-9] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [213] milliseconds.
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-10] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [207] milliseconds.
2019-02-11 16:41:01,320 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2019-02-11 16:41:42,963 WARN [http-nio-7990-exec-5] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:43:46,374 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2019-02-11 16:44:49,975 WARN [spring-startup] com.hazelcast.instance.Node [192.168.100.165]:5701 [atlbitbucket] [3.7.4-atlassian-43] No join method is enabled! Starting standalone.
2019-02-11 16:44:52,885 WARN [http-nio-7990-exec-10] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:48:43,883 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:51:49,928 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:54:44,778 WARN [http-nio-7990-exec-9] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:54:57,901 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:38168
2019-02-11 16:56:04,461 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:41183
2019-02-11 16:58:21,722 WARN [http-nio-7990-exec-10] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:21,737 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:21,792 WARN [http-nio-7990-exec-4] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:25,184 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:25,206 WARN [http-nio-7990-exec-8] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:25,242 WARN [http-nio-7990-exec-9] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:30,216 WARN [http-nio-7990-exec-3] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:30,288 WARN [http-nio-7990-exec-5] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:30,439 WARN [http-nio-7990-exec-1] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:34,135 WARN [http-nio-7990-exec-2] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:34,219 WARN [http-nio-7990-exec-1] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:34,363 WARN [http-nio-7990-exec-8] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:44,982 WARN [alert-dispatch:thread-1] atlassian-diagnostics 2019-02-12T01:02:26.003Z Component 'diagnostics.event.name' alerted 'diagnostics.event.issue.2001.summary' (details: {"timeMillis":16624,"eventType":"com.atlassian.plugin.event.events.PluginEnabledEvent"}, trigger: {"pluginKey": "System", "pluginVersion": "5.16.0", "module": "com.atlassian.stash.internal.i18n.PluginI18nService"})
2019-02-11 17:04:12,135 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2019-02-11 17:04:59,462 WARN [alert-dispatch:thread-1] atlassian-diagnostics 2019-02-12T01:04:28.872Z Component 'Event system' alerted 'Slow event listener detected' (details: {"timeMillis":28345,"eventType":"com.atlassian.plugin.event.events.PluginEnabledEvent"}, trigger: {"pluginKey": "System", "pluginVersion": "5.16.0", "module": "com.atlassian.stash.internal.i18n.PluginI18nService"})
2019-02-11 17:05:01,542 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2019-02-11 17:05:01,921 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2019-02-11 17:05:02,450 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2019-02-11 17:05:06,190 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2019-02-11 17:05:08,959 INFO [spring-startup] c.a.b.i.ssh.server.SshServerFactory Using built in SSHD NIO service factory
2019-02-11 17:05:09,632 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2019-02-11 17:05:17,721 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2019-02-11 17:05:20,519 INFO [spring-startup] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2019-02-11 17:05:20,525 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2019-02-11 17:05:20,531 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2019-02-11 17:05:20,953 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2019-02-11 17:05:21,399 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2019-02-11 17:05:22,757 WARN [UpmScheduler:thread-1] com.atlassian.upm.pac.PacClientImpl Update check request may take longer because of the number of apps
2019-02-11 17:05:24,126 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/local/bin/git version 2.20.1
2019-02-11 17:05:24,966 INFO [spring-startup] c.a.s.i.hazelcast.HazelcastLifecycle Updating Hazelcast instance capabilities
2019-02-11 17:05:30,675 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Attempting provisioning in startup job
2019-02-11 17:05:30,722 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2019-02-11 17:05:31,392 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 3 search health checks
2019-02-11 17:05:36,262 INFO [I/O dispatcher 21] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2019-02-11 17:05:36,931 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2019-02-11 17:07:01,663 WARN [http-nio-7990-exec-2] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:07:01,748 WARN [http-nio-7990-exec-7] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:07:01,892 WARN [http-nio-7990-exec-8] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:10:36,902 INFO [Caesium-1-4] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 2 search health checks
2019-02-11 17:10:37,222 INFO [I/O dispatcher 41] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2019-02-11 17:10:37,537 INFO [Caesium-1-4] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2019-02-11 17:15:37,544 INFO [Caesium-1-3] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 2 search health checks
2019-02-11 17:15:37,972 INFO [I/O dispatcher 61] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2019-02-11 17:15:38,444 INFO [Caesium-1-3] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT10M

[Stephen Sifers]

Hello Daniel and welcome to the Community!

Thank you for providing your startup logs for Bitbucket.

Could you please provide some further information on your Bitbucket instance:

1. What OS are you running?
2. What CPU and how many cores/threads are allocated?
3. How much memory is allocated to the machine?
4. What type is Disk is used for the machine (SAS,SATA,SSD)?
5. What database are you using?

Something to review would be your monitoring tools which are monitoring your Bitbucket server. If you’re using a monitoring solution, please review the CPU, Memory and Disk load during your startup to see if you’re able to see if a hardware limitation is present.

We look forward to your response so we may assist with finding a cause to your slow load times.

Regards.
Stephen Sifers