what causes Hazelcast instance to become inactive?

A team member was doing some load testing and Stash became unresponsive.  According to the log Hazelcast was not active.  I stopped and restarted Stash and it seems fine now.  What would cause this to happen and can we make any configuration change to prevent it?

Here is a portion of the log

ava.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_75]
at java.util.concurrent.FutureTask.get(FutureTask.java:202) ~[na:1.7.0_75]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator.authenticate(DefaultPublicKeyAuthenticator.java:93) ~[stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.UserAuthPublicKey.doAuth(UserAuthPublicKey.java:72) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.AbstractUserAuth.auth(AbstractUserAuth.java:49) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.session.ServerUserAuthService.process(ServerUserAuthService.java:145) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:405) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:301) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:738) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:102) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [stash-ssh-3.7.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
... 1 frame trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1056) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.distributed.IMapRegionCache.get(IMapRegionCache.java:75) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.access.AbstractAccessDelegate.get(AbstractAccessDelegate.java:85) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.region.EntityRegionAccessStrategyAdapter.get(EntityRegionAccessStrategyAdapter.java:56) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:55) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:67) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:598) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:452) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.load(CrowdAwareLoadListener.java:104) ~[stash-dao-impl-3.7.1.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2551) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:86) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:24) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:219) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:213) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:346) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:336) ~[na:na]

at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.access$300(DefaultSshKeyService.java:33) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:158) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25) ~[sal-core-2.13.3.jar:na]
at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:27) ~[stash-platform-3.7.1.jar:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21) ~[sal-core-2.13.3.jar:na]
at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:48) ~[na:na]
at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:19) ~[na:na]
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:276) ~[na:na]
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.executeInTransaction(TenantAwareActiveObjects.java:383) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.getByPublicKey(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:84) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:75) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]
... 3 common frames omitted
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,868 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,869 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!

 

1 answer

This widget could not be displayed.

Hey Mark,

Sometimes the error "Hazelcast instance is not active" is shown when Stash is not available. By any changes you are seeing any out of memory errors in stash?

java.lang.OutOfMemoryError: PermGen space

 

If you find that in your logs then try to increase stahs memory as described in below KB:

https://confluence.atlassian.com/display/STASHKB/Stash+Crashes+Due+to+java.lang.OutOfMemoryError+PermGen+Space+Error

Hope it helps.
Thanks and Regards,
David| The Engineer that like Windows for realsies.

Not exactly that message but I do see this one java.lang.OutOfMemoryError: Java heap space In a previous installation I was getting java.lang.OutOfMemoryError: GC overhead limit exceeded and increasing these seemed to help: JVM_MINIMUM_MEMORY="512m" JVM_MAXIMUM_MEMORY="768m" But is Garbage Collection and Java heap space the same memory? I don't want to veer from the standard unless it is warranted..

Yes they are, increase the minimum and maximum heap to 1024 so we will avoid the memory variation and hence reduce the chances of the out of memory error to happens again.

With the GC errors we went through several increases and so I am moving it to 2048. Thanks

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Posted Aug 10, 2018 in Bitbucket

What is your #1 piece of advice to new users getting started using Bitbucket?

Hello Community! My name is Claire Maynard, and I’m a Product Marketing Manager on the Bitbucket team. I’m interested in hearing what advice or tips you have for new users getting started on ...

244 views 14 2
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you