A team member was doing some load testing and Stash became unresponsive. According to the log Hazelcast was not active. I stopped and restarted Stash and it seems fine now. What would cause this to happen and can we make any configuration change to prevent it?
Here is a portion of the log
ava.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_75]
at java.util.concurrent.FutureTask.get(FutureTask.java:202) ~[na:1.7.0_75]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator.authenticate(DefaultPublicKeyAuthenticator.java:93) ~[stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.UserAuthPublicKey.doAuth(UserAuthPublicKey.java:72) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.auth.AbstractUserAuth.auth(AbstractUserAuth.java:49) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.server.session.ServerUserAuthService.process(ServerUserAuthService.java:145) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:405) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:301) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:738) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [stash-ssh-3.7.1.jar:na]
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:102) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121) [stash-ssh-3.7.1.jar:na]
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [stash-ssh-3.7.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
... 1 frame trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1056) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83) ~[hazelcast-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.distributed.IMapRegionCache.get(IMapRegionCache.java:75) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.access.AbstractAccessDelegate.get(AbstractAccessDelegate.java:85) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at com.hazelcast.hibernate.region.EntityRegionAccessStrategyAdapter.get(EntityRegionAccessStrategyAdapter.java:56) ~[hazelcast-hibernate4-3.3.1-atlassian-10.jar:3.3.1-atlassian-10]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:55) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:67) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:598) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:452) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.load(CrowdAwareLoadListener.java:104) ~[stash-dao-impl-3.7.1.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2551) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:86) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:24) ~[stash-dao-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:219) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:213) ~[stash-service-impl-3.7.1.jar:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:346) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.initialize(DefaultSshKeyService.java:336) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.access$300(DefaultSshKeyService.java:33) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:158) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService$3.doInTransaction(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25) ~[sal-core-2.13.3.jar:na]
at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:27) ~[stash-platform-3.7.1.jar:na]
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21) ~[sal-core-2.13.3.jar:na]
at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:48) ~[na:na]
at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:19) ~[na:na]
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:276) ~[na:na]
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.executeInTransaction(TenantAwareActiveObjects.java:383) ~[na:na]
at com.atlassian.stash.internal.key.ssh.DefaultSshKeyService.getByPublicKey(DefaultSshKeyService.java:154) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:84) ~[na:na]
at com.atlassian.stash.internal.ssh.server.DefaultPublicKeyAuthenticator$1.call(DefaultPublicKeyAuthenticator.java:75) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]
... 3 common frames omitted
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,865 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,867 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,868 WARN [AtlassianEvent::thread-7] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
2015-04-02 18:51:31,869 WARN [AtlassianEvent::thread-17] c.a.a.c.c.AnalyticsConfig Couldn't check the analytics settings. This can safely be ignored during plugin shutdown. Detail: Hazelcast instance is not active!
