I am getting this error when Bitbucket tries to start up.
2020-05-27 17:14:54,931 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.4.RELEASE)
2020-05-27 17:14:55,289 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on ip-172-31-33-69 with PID 7105 (/opt/atlassian/bitbucket/7.1.1/app/WEB-INF/classes started by atlbitbucket in /opt/atlassian/bitbucket/7.1.1/bin)
2020-05-27 17:14:55,294 INFO [main] c.a.b.i.b.BitbucketServerApplication The following profiles are active: nio
2020-05-27 17:14:56,890 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 7.1.1 (3342f37 built on Wed Apr 08 05:20:52 UTC 2020)
2020-05-27 17:14:56,890 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: AdoptOpenJDK OpenJDK 64-Bit Server VM 1.8.0_242-b08
2020-05-27 17:14:59,890 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 8.34 seconds (JVM running for 10.985)
2020-05-27 17:15:05,752 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2020-05-27 17:15:13,143 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2020-05-27 17:15:15,712 WARN [spring-startup] com.hazelcast.instance.Node [172.31.33.69]:5701 [atlbitbucket] [3.12.6] No join method is enabled! Starting standalone.
2020-05-27 17:15:36,732 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:55695
2020-05-27 17:15:39,828 ERROR [hz.hazelcast.scheduled.thread-] com.hazelcast.spi.ExecutionService [172.31.33.69]:5701 [atlbitbucket] [3.12.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@28930c5e
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate.addNewWorkerIfRequired(CachedExecutorServiceDelegate.java:146)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate.execute(CachedExecutorServiceDelegate.java:116)
at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:52)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
... 1 frame trimmed
2020-05-27 17:18:54,489 INFO [main] o.s.boot.SpringApplication
This is on an ubuntu instance. I am trying to run jstack to help track the issue but jstack doesn't work.
((This is the article https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/ ))
jstack -l 1488 jstackoutput.txt
Attaching to core jstackoutput.txt from executable 1488, please wait...
Error attaching to core file: Can't attach to the core file
Not sure what to do to track the issue. Would appreciate any help here.
The previous message was in the /var/atlassian/application-data/bitbucket/log/atlassian-bitbuck-2020-05-27.log file. The below is from the atlassian-bitbucket.log file from the same location.
2020-05-28 03:27:14,206 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,241 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,241 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,692 WARN [http-nio-7990-exec-3] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,692 WARN [http-nio-7990-exec-3] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,808 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
The below is from launcher.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f007ee70000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid10669.log
#
# Compiler replay data is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/replay_pid10669.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000101133000, 131072, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 131072 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid7105.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1c6b6fd000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid7433.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1c695ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 139760004671232 also had an error]
2020-05-27 17:27:01
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):
"SpringContextShutdownHook" #29 prio=5 os_prio=0 tid=0x00007fce4004b800 nid=0x1ed7 in Object.wait() [0x00007fce293e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c0def1f0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at com.atlassian.johnson.spring.lifecycle.LifecycleDispatcherServlet.destroy(LifecycleDispatcherServlet.java:55)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1291)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1592)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5389)
- locked <0x00000000c0afc2f8> (a org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afc2f8> (a org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
- locked <0x00000000c0afc678> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afc678> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:473)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afd268> (a org.apache.catalina.core.StandardService)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:992)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c04973e0> (a org.apache.catalina.core.StandardServer)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:478)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stopTomcat(TomcatWebServer.java:269)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stop(TomcatWebServer.java:324)
- locked <0x00000000c0921960> (a java.lang.Object)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.stopAndReleaseWebServer(ServletWebServerApplicationContext.java:306)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onClose(ServletWebServerApplicationContext.java:172)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1035)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:948)
- locked <0x00000000c0925d50> (a java.lang.Object)
"SIGTERM handler" #37 daemon prio=9 os_prio=0 tid=0x00007fce2c002800 nid=0x1ed4 in Object.wait() [0x00007fce29bec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c0fa4e30> (a org.springframework.context.support.AbstractApplicationContext$1)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000c0423428> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Here is a screenshot showing memory usage using top command.
https://prnt.sc/sp55h0
