We came in this morning to a non-responsive Bibucket server. Restarting, it is taking over 30 minutes to complete a start up and become available.
2019-02-11 16:37:14,896 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.17.RELEASE)
2019-02-11 16:37:22,268 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on sc-livemedia2.sowingcircle.net with PID 7133 (/opt/atlassian/bitbucket/5.16.0/app/WEB-INF/classes started by atlbitbucket in /)
2019-02-11 16:37:22,278 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2019-02-11 16:37:59,232 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.16.0 (c03f0a7 built on Thu Nov 15 20:10:50 PST 2018)
2019-02-11 16:37:59,236 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_191-b12
2019-02-11 16:39:01,017 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 135.944 seconds (JVM running for 217.121)
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-2] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [208] milliseconds.
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-8] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [210] milliseconds.
2019-02-11 16:39:40,775 WARN [http-nio-7990-exec-7] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [209] milliseconds.
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-9] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [213] milliseconds.
2019-02-11 16:39:40,770 WARN [http-nio-7990-exec-10] o.a.c.util.SessionIdGeneratorBase Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [207] milliseconds.
2019-02-11 16:41:01,320 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2019-02-11 16:41:42,963 WARN [http-nio-7990-exec-5] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:43:46,374 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2019-02-11 16:44:49,975 WARN [spring-startup] com.hazelcast.instance.Node [192.168.100.165]:5701 [atlbitbucket] [3.7.4-atlassian-43] No join method is enabled! Starting standalone.
2019-02-11 16:44:52,885 WARN [http-nio-7990-exec-10] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:48:43,883 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:51:49,928 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:54:44,778 WARN [http-nio-7990-exec-9] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [scmServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:54:57,901 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:38168
2019-02-11 16:56:04,461 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:41183
2019-02-11 16:58:21,722 WARN [http-nio-7990-exec-10] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:21,737 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:21,792 WARN [http-nio-7990-exec-4] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:25,184 WARN [http-nio-7990-exec-6] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:25,206 WARN [http-nio-7990-exec-8] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 16:58:25,242 WARN [http-nio-7990-exec-9] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:30,216 WARN [http-nio-7990-exec-3] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:30,288 WARN [http-nio-7990-exec-5] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:30,439 WARN [http-nio-7990-exec-1] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:34,135 WARN [http-nio-7990-exec-2] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:34,219 WARN [http-nio-7990-exec-1] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:34,363 WARN [http-nio-7990-exec-8] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:02:44,982 WARN [alert-dispatch:thread-1] atlassian-diagnostics 2019-02-12T01:02:26.003Z Component 'diagnostics.event.name' alerted 'diagnostics.event.issue.2001.summary' (details: {"timeMillis":16624,"eventType":"com.atlassian.plugin.event.events.PluginEnabledEvent"}, trigger: {"pluginKey": "System", "pluginVersion": "5.16.0", "module": "com.atlassian.stash.internal.i18n.PluginI18nService"})
2019-02-11 17:04:12,135 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2019-02-11 17:04:59,462 WARN [alert-dispatch:thread-1] atlassian-diagnostics 2019-02-12T01:04:28.872Z Component 'Event system' alerted 'Slow event listener detected' (details: {"timeMillis":28345,"eventType":"com.atlassian.plugin.event.events.PluginEnabledEvent"}, trigger: {"pluginKey": "System", "pluginVersion": "5.16.0", "module": "com.atlassian.stash.internal.i18n.PluginI18nService"})
2019-02-11 17:05:01,542 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2019-02-11 17:05:01,921 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2019-02-11 17:05:02,450 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2019-02-11 17:05:06,190 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2019-02-11 17:05:08,959 INFO [spring-startup] c.a.b.i.ssh.server.SshServerFactory Using built in SSHD NIO service factory
2019-02-11 17:05:09,632 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2019-02-11 17:05:17,721 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2019-02-11 17:05:20,519 INFO [spring-startup] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2019-02-11 17:05:20,525 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2019-02-11 17:05:20,531 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2019-02-11 17:05:20,953 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2019-02-11 17:05:21,399 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2019-02-11 17:05:22,757 WARN [UpmScheduler:thread-1] com.atlassian.upm.pac.PacClientImpl Update check request may take longer because of the number of apps
2019-02-11 17:05:24,126 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/local/bin/git version 2.20.1
2019-02-11 17:05:24,966 INFO [spring-startup] c.a.s.i.hazelcast.HazelcastLifecycle Updating Hazelcast instance capabilities
2019-02-11 17:05:30,675 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Attempting provisioning in startup job
2019-02-11 17:05:30,722 INFO [Caesium-1-3] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2019-02-11 17:05:31,392 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 3 search health checks
2019-02-11 17:05:36,262 INFO [I/O dispatcher 21] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2019-02-11 17:05:36,931 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2019-02-11 17:07:01,663 WARN [http-nio-7990-exec-2] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:07:01,748 WARN [http-nio-7990-exec-7] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:07:01,892 WARN [http-nio-7990-exec-8] c.a.j.s.w.c.s.JohnsonHttpRequestHandlerServlet HttpRequestHandlerServlet [fileServerServlet] cannot be initialised to service an incoming request; redirecting to /unavailable
2019-02-11 17:10:36,902 INFO [Caesium-1-4] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 2 search health checks
2019-02-11 17:10:37,222 INFO [I/O dispatcher 41] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2019-02-11 17:10:37,537 INFO [Caesium-1-4] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2019-02-11 17:15:37,544 INFO [Caesium-1-3] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 2 search health checks
2019-02-11 17:15:37,972 INFO [I/O dispatcher 61] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2019-02-11 17:15:38,444 INFO [Caesium-1-3] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT10M
