Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

What to do when it states java.lang.OutOfMemoryError: unable to create new native thread

Edited

I am getting this error when Bitbucket tries to start up. 

 

2020-05-27 17:14:54,931 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.4.RELEASE)


2020-05-27 17:14:55,289 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on ip-172-31-33-69 with PID 7105 (/opt/atlassian/bitbucket/7.1.1/app/WEB-INF/classes started by atlbitbucket in /opt/atlassian/bitbucket/7.1.1/bin)
2020-05-27 17:14:55,294 INFO [main] c.a.b.i.b.BitbucketServerApplication The following profiles are active: nio
2020-05-27 17:14:56,890 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 7.1.1 (3342f37 built on Wed Apr 08 05:20:52 UTC 2020)
2020-05-27 17:14:56,890 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: AdoptOpenJDK OpenJDK 64-Bit Server VM 1.8.0_242-b08
2020-05-27 17:14:59,890 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 8.34 seconds (JVM running for 10.985)
2020-05-27 17:15:05,752 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2020-05-27 17:15:13,143 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2020-05-27 17:15:15,712 WARN [spring-startup] com.hazelcast.instance.Node [172.31.33.69]:5701 [atlbitbucket] [3.12.6] No join method is enabled! Starting standalone.
2020-05-27 17:15:36,732 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:55695
2020-05-27 17:15:39,828 ERROR [hz.hazelcast.scheduled.thread-] com.hazelcast.spi.ExecutionService [172.31.33.69]:5701 [atlbitbucket] [3.12.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@28930c5e
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate.addNewWorkerIfRequired(CachedExecutorServiceDelegate.java:146)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate.execute(CachedExecutorServiceDelegate.java:116)
at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:52)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
... 1 frame trimmed
2020-05-27 17:18:54,489 INFO [main] o.s.boot.SpringApplication

 

 

 

 

This is on an ubuntu instance. I am trying to run jstack to help track the issue but jstack doesn't work.  
((This is the article https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/  ))

jstack -l 1488 jstackoutput.txt
Attaching to core jstackoutput.txt from executable 1488, please wait...
Error attaching to core file: Can't attach to the core file

Not sure what to do to track the issue. Would appreciate any help here.

 

 

The previous message was in the /var/atlassian/application-data/bitbucket/log/atlassian-bitbuck-2020-05-27.log file. The below is from the atlassian-bitbucket.log file from the same location. 

 

2020-05-28 03:27:14,206 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,241 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,241 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,692 WARN [http-nio-7990-exec-3] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,692 WARN [http-nio-7990-exec-3] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,808 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available

 

The below is from launcher.log

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f007ee70000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid10669.log
#
# Compiler replay data is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/replay_pid10669.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000101133000, 131072, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 131072 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid7105.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1c6b6fd000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid7433.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1c695ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 139760004671232 also had an error]
2020-05-27 17:27:01
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):

"SpringContextShutdownHook" #29 prio=5 os_prio=0 tid=0x00007fce4004b800 nid=0x1ed7 in Object.wait() [0x00007fce293e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c0def1f0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at com.atlassian.johnson.spring.lifecycle.LifecycleDispatcherServlet.destroy(LifecycleDispatcherServlet.java:55)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1291)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1592)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5389)
- locked <0x00000000c0afc2f8> (a org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afc2f8> (a org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
- locked <0x00000000c0afc678> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afc678> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)


at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:473)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afd268> (a org.apache.catalina.core.StandardService)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:992)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c04973e0> (a org.apache.catalina.core.StandardServer)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:478)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stopTomcat(TomcatWebServer.java:269)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stop(TomcatWebServer.java:324)
- locked <0x00000000c0921960> (a java.lang.Object)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.stopAndReleaseWebServer(ServletWebServerApplicationContext.java:306)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onClose(ServletWebServerApplicationContext.java:172)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1035)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:948)
- locked <0x00000000c0925d50> (a java.lang.Object)

"SIGTERM handler" #37 daemon prio=9 os_prio=0 tid=0x00007fce2c002800 nid=0x1ed4 in Object.wait() [0x00007fce29bec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c0fa4e30> (a org.springframework.context.support.AbstractApplicationContext$1)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000c0423428> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m

 

Here is a screenshot showing memory usage using top command. 

https://prnt.sc/sp55h0

1 answer

0 votes

Hi Frank,

If it's unable to create a new native thread it means the system is actually out of physical RAM and not out of heap, so running jstack won't help.

How much RAM do you have on the machine and what is the size of the heap that you have configured?

Cheers,

Christian

Premier Support Engineer

Atlassian

We increased the instance size to one with double the memory and the issue remained. The fix was to manually assign the project directory in the bitbucket settings file. Afterword it seems to be working mostly, though we are having issues getting a sql server installed for it to use. If it isn't one thing it is something else. Thanks!

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

3,711 views 10 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you