what causes Hazelcast instance to become inactive?

A team member was doing some load testing and Stash became unresponsive.  According to the log Hazelcast was not active.  I stopped and restarted Stash and it seems fine now.  What would cause this to happen and can we make any configuration change to prevent it?

Here is a portion of the log

ava.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_75]
at java.util.concurrent.FutureTask.get(FutureTask.java:202) ~[na:1.7.0_75]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator.authenticate(DefaultPublicKeyAuthenticator.java:93) ~[stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.UserAuthPublicKey.doAuth(UserAuthPublicKey.java:72) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.AbstractUserAuth.auth(AbstractUserAuth.java:49) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.session.ServerUserAuthService.process(ServerUserAuthService.java:145) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:405) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:301) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:738) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:102) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [stash-ssh-3.7.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
... 1 frame trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1056) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.distributed.IMapRegionCache.get(IMapRegionCache.java:75) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.access.AbstractAccessDelegate.get(AbstractAccessDelegate.java:85) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.region.EntityRegionAccessStrategyAdapter.get(EntityRegionAccessStrategyAdapter.java:56) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:55) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:67) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:598) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:452) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.load(CrowdAwareLoadListener.java:104) ~[stash-dao-impl-3.7.1.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2551) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:86) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:24) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:219) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:213) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:346) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:336) ~[na:na]

at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.access$300(DefaultSshKeyService.java:33) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:158) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25) ~[sal-core-2.13.3.jar:na]
at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:27) ~[stash-platform-3.7.1.jar:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21) ~[sal-core-2.13.3.jar:na]
at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:48) ~[na:na]
at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:19) ~[na:na]
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:276) ~[na:na]
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.executeInTransaction(TenantAwareActiveObjects.java:383) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.getByPublicKey(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:84) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:75) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]
... 3 common frames omitted
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,868 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,869 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!

 

1 answer

Hey Mark,

Sometimes the error "Hazelcast instance is not active" is shown when Stash is not available. By any changes you are seeing any out of memory errors in stash?

java.lang.OutOfMemoryError: PermGen space

 

If you find that in your logs then try to increase stahs memory as described in below KB:

https://confluence.atlassian.com/display/STASHKB/Stash+Crashes+Due+to+java.lang.OutOfMemoryError+PermGen+Space+Error

Hope it helps.
Thanks and Regards,
David| The Engineer that like Windows for realsies.

Not exactly that message but I do see this one java.lang.OutOfMemoryError: Java heap space In a previous installation I was getting java.lang.OutOfMemoryError: GC overhead limit exceeded and increasing these seemed to help: JVM_MINIMUM_MEMORY="512m" JVM_MAXIMUM_MEMORY="768m" But is Garbage Collection and Java heap space the same memory? I don't want to veer from the standard unless it is warranted..

Yes they are, increase the minimum and maximum heap to 1024 so we will avoid the memory variation and hence reduce the chances of the out of memory error to happens again.

With the GC errors we went through several increases and so I am moving it to 2048. Thanks

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jun 06, 2018 in Bitbucket

Do you use Bitbucket Cloud and Jira Cloud? If so, let us know!

Hi Community, I'm Julia and I'm on the Jira Software Cloud marketing team!  We're looking for companies or teams using Bitbucket Cloud and Jira Software Cloud. If your team fits the t...

161 views 6 3
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you