what causes Hazelcast instance to become inactive?

Mark Ruedy April 2, 2015

A team member was doing some load testing and Stash became unresponsive.  According to the log Hazelcast was not active.  I stopped and restarted Stash and it seems fine now.  What would cause this to happen and can we make any configuration change to prevent it?

Here is a portion of the log

ava.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_75]
at java.util.concurrent.FutureTask.get(FutureTask.java:202) ~[na:1.7.0_75]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator.authenticate(DefaultPublicKeyAuthenticator.java:93) ~[stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.UserAuthPublicKey.doAuth(UserAuthPublicKey.java:72) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.AbstractUserAuth.auth(AbstractUserAuth.java:49) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.session.ServerUserAuthService.process(ServerUserAuthService.java:145) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:405) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:301) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:738) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:102) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [stash-ssh-3.7.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
... 1 frame trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1056) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.distributed.IMapRegionCache.get(IMapRegionCache.java:75) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.access.AbstractAccessDelegate.get(AbstractAccessDelegate.java:85) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.region.EntityRegionAccessStrategyAdapter.get(EntityRegionAccessStrategyAdapter.java:56) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:55) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:67) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:598) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:452) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.load(CrowdAwareLoadListener.java:104) ~[stash-dao-impl-3.7.1.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2551) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:86) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:24) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:219) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:213) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:346) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:336) ~[na:na]

at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.access$300(DefaultSshKeyService.java:33) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:158) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25) ~[sal-core-2.13.3.jar:na]
at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:27) ~[stash-platform-3.7.1.jar:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21) ~[sal-core-2.13.3.jar:na]
at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:48) ~[na:na]
at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:19) ~[na:na]
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:276) ~[na:na]
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.executeInTransaction(TenantAwareActiveObjects.java:383) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.getByPublicKey(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:84) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:75) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]
... 3 common frames omitted
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,868 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,869 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!

 

1 answer

1 vote
Deividi Luvison
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 3, 2015

Hey Mark,

Sometimes the error "Hazelcast instance is not active" is shown when Stash is not available. By any changes you are seeing any out of memory errors in stash?

java.lang.OutOfMemoryError: PermGen space

 

If you find that in your logs then try to increase stahs memory as described in below KB:

https://confluence.atlassian.com/display/STASHKB/Stash+Crashes+Due+to+java.lang.OutOfMemoryError+PermGen+Space+Error

Hope it helps.
Thanks and Regards,
David| The Engineer that like Windows for realsies.
Mark Ruedy April 3, 2015

Not exactly that message but I do see this one java.lang.OutOfMemoryError: Java heap space In a previous installation I was getting java.lang.OutOfMemoryError: GC overhead limit exceeded and increasing these seemed to help: JVM_MINIMUM_MEMORY="512m" JVM_MAXIMUM_MEMORY="768m" But is Garbage Collection and Java heap space the same memory? I don't want to veer from the standard unless it is warranted..

Deividi Luvison
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 3, 2015

Yes they are, increase the minimum and maximum heap to 1024 so we will avoid the memory variation and hence reduce the chances of the out of memory error to happens again.

Mark Ruedy April 3, 2015

With the GC errors we went through several increases and so I am moving it to 2048. Thanks

Deividi Luvison
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 3, 2015

You' ve welcome :)

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events