Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,362,012
Community Members
 
Community Events
168
Community Groups

what causes Hazelcast instance to become inactive?

A team member was doing some load testing and Stash became unresponsive.  According to the log Hazelcast was not active.  I stopped and restarted Stash and it seems fine now.  What would cause this to happen and can we make any configuration change to prevent it?

Here is a portion of the log

ava.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_75]
at java.util.concurrent.FutureTask.get(FutureTask.java:202) ~[na:1.7.0_75]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator.authenticate(DefaultPublicKeyAuthenticator.java:93) ~[stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.UserAuthPublicKey.doAuth(UserAuthPublicKey.java:72) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.AbstractUserAuth.auth(AbstractUserAuth.java:49) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.session.ServerUserAuthService.process(ServerUserAuthService.java:145) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:405) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:301) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:738) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:102) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [stash-ssh-3.7.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
... 1 frame trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1056) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.distributed.IMapRegionCache.get(IMapRegionCache.java:75) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.access.AbstractAccessDelegate.get(AbstractAccessDelegate.java:85) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.region.EntityRegionAccessStrategyAdapter.get(EntityRegionAccessStrategyAdapter.java:56) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:55) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:67) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:598) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:452) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.load(CrowdAwareLoadListener.java:104) ~[stash-dao-impl-3.7.1.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2551) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:86) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:24) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:219) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:213) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:346) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:336) ~[na:na]

at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.access$300(DefaultSshKeyService.java:33) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:158) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25) ~[sal-core-2.13.3.jar:na]
at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:27) ~[stash-platform-3.7.1.jar:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21) ~[sal-core-2.13.3.jar:na]
at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:48) ~[na:na]
at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:19) ~[na:na]
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:276) ~[na:na]
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.executeInTransaction(TenantAwareActiveObjects.java:383) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.getByPublicKey(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:84) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:75) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]
... 3 common frames omitted
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,868 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,869 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!

 

1 answer

1 vote

Hey Mark,

Sometimes the error "Hazelcast instance is not active" is shown when Stash is not available. By any changes you are seeing any out of memory errors in stash?

java.lang.OutOfMemoryError: PermGen space

 

If you find that in your logs then try to increase stahs memory as described in below KB:

https://confluence.atlassian.com/display/STASHKB/Stash+Crashes+Due+to+java.lang.OutOfMemoryError+PermGen+Space+Error

Hope it helps.
Thanks and Regards,
David| The Engineer that like Windows for realsies.

Not exactly that message but I do see this one java.lang.OutOfMemoryError: Java heap space In a previous installation I was getting java.lang.OutOfMemoryError: GC overhead limit exceeded and increasing these seemed to help: JVM_MINIMUM_MEMORY="512m" JVM_MAXIMUM_MEMORY="768m" But is Garbage Collection and Java heap space the same memory? I don't want to veer from the standard unless it is warranted..

Yes they are, increase the minimum and maximum heap to 1024 so we will avoid the memory variation and hence reduce the chances of the out of memory error to happens again.

With the GC errors we went through several increases and so I am moving it to 2048. Thanks

You' ve welcome :)

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Git push size limits are coming to Bitbucket Cloud starting April 4th, 2022

Beginning on April 4th, we will be implementing push limits. This means that your push cannot be completed if it is over 3.5 GB. If you do attempt to complete a push that is over 3.5 GB, it will fail...

3,503 views 3 10
Read article

Atlassian Community Events