Stash starts but hangs forever

Question

Hello,

I tested Stash 3.7.0 on my local machine and using the default install connecting to localhost:7990 works. Once I installed Stash 3.7.0 on a server and connected to my.domain:7990 it looks like it hangs on "connecting to my.domain:7990". I run stash in the foreground and here's what I see.

EDIT: On the same server I am successfully running an instance of JIRA, Confluence, and Crucible.

[root@s1 bin]# ./start-stash.sh -fg
Starting Atlassian Stash as current user

If you do not see a 'Server startup' message within 3 minutes, please see the troubleshooting guide at:

https://confluence.atlassian.com/display/STASHKB/Troubleshooting+Installation

-------------------------------------------------------------------------------
Stash is being run with a umask that contains potentially unsafe settings.
The following issues were found with the mask "u=rwx,g=rx,o=rx" (0022):
 - access is allowed to 'others'. It is recommended that 'others' be denied
   all access for security reasons.

The recommended umask for Stash is "u=,g=w,o=rwx" (0027) and can be
configured in setenv.sh
-------------------------------------------------------------------------------
Detecting JVM PermGen support...
PermGen switch is supported. Setting to 256m

Using STASH_HOME:      /home/corp/atlassian/application-data/stash
Using CATALINA_BASE:   /home/corp/atlassian/stash/3.7.0
Using CATALINA_HOME:   /home/corp/atlassian/stash/3.7.0
Using CATALINA_TMPDIR: /home/corp/atlassian/stash/3.7.0/temp
Using JRE_HOME:        /home/corp/atlassian-jira-4.3.4-standalone/jre
Using CLASSPATH:       /home/corp/atlassian/stash/3.7.0/bin/bootstrap.jar:/home/corp/atlassian/stash/3.7.0/bin/tomcat-juli.jar
Using CATALINA_PID:    /home/corp/atlassian/stash/3.7.0/work/catalina.pid
06-Mar-2015 11:05:47.018 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
06-Mar-2015 11:05:47.019 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
06-Mar-2015 11:05:47.068 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /home/corp/atlassian/stash/3.7.0/lib/native:/home/corp/atlassian/application-data/stash/lib/native
06-Mar-2015 11:05:47.234 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-7990"]
06-Mar-2015 11:05:47.260 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
06-Mar-2015 11:05:47.264 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 841 ms
06-Mar-2015 11:05:47.294 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
06-Mar-2015 11:05:47.294 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.15
06-Mar-2015 11:06:01.791 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-7990"]
06-Mar-2015 11:06:01.809 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 14545 ms

cirovladimir · Answer

I'm sorry to bring back this old issue, but I just had the same problem described here, with no apparent error found in the logs nor elsewhere.

In our case it was due to a corrupted database caused by a server getting out of space and producing the error `FATAL: btree level 1 not found in index "19881"` which is [explained in more detail here](https://cirovladimir.wordpress.com/2018/12/04/postgres-btree-level-1-not-found-in-index/). After recovery with the pg_resetxlog some index errors were found and fixed by recreating the indexes with REINDEX (https://www.postgresql.org/docs/9.4/sql-reindex.html). We were able to restore the jira instance but the stash instance kept freezing at Initializing Spring context. I found that probably the cause would be a database lock which otherwise would be solved by issuing a `UPDATE databasechangeloglock SET LOCKED=false, LOCKGRANTED=null, LOCKEDBY=null where ID=1;` (https://stackoverflow.com/questions/54827560/bitbucket-migrating-home-directory-and-fails-to-start). I tried the aformentioned sql command but it got stuck, I even tried to kill sessions identified by `SELECT * FROM pg_stat_activity WHERE state = 'active';` and killed by `SELECT pg_terminate_backend(PID);`, restarted the database and reattempted the update but got stuck again. Now I was sure it was the culprit of stash not starting.

To solve this problem, we had to make a backup with pg_dumpall and then restore to a new instance (which will recreate all objects in the database and then import the data).

```
sudo su postgres
pg_dumpall > backup.dmp
exit
sudo service postgresql stop
mkdir -p /tmp/postgres
sudo chown postgres /tmp/postgres
sudo su postgres
cd /tmp/postgres
/usr/lib/postgresql/9.3/bin/initdb .
/usr/lib/postgresql/9.3/bin/postgres -D /tmp/postgres
```
in a new terminal restore the database
```
sudo su postgres
psql -f backup.dmp postgres
```
and finally restart jira and stash

```
sudo service jira start
sudo service atlstash start
```

Once restored, we're starting to upgrade our database and atlassian applications :P

Michael Heemskerk · Answer

Hmmm... that does not look like it should. The output you've posted is what I would expect on the commandline where you started Stash (with the -fg option). The atlassian-stash.log file should contain something like:

2015-03-20 14:58:16,817 INFO  [spring-startup]  c.a.s.i.server.BuildInfoLogger Starting Stash 3.8.0-SNAPSHOT (${commi built on Fri Mar 20 14:58:16 CET 2015)
2015-03-20 14:58:16,822 INFO  [spring-startup]  c.a.s.i.server.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0-b132
2015-03-20 14:58:16,843 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /Users/mheemskerk/inst/clean
2015-03-20 14:58:18,865 WARN  [spring-startup]  com.jolbox.bonecp.BoneCPConfig JDBC password not set in driver properties, copying it from pool config
2015-03-20 14:58:20,892 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /Users/mheemskerk/inst/clean/shared
2015-03-20 14:58:45,988 INFO  [spring-startup]  c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:61884
2015-03-20 14:58:52,675 WARN  [spring-startup]  c.a.p.o.f.t.s.ScanDescriptorForHostClassesStage The plugin 'atlassian-universal-plugin-manager-plugin-2.18.2.jar' uses a package 'org.apache.commons.fileupload.servlet' that is also exported by the application.  It is highly recommended that the plugin use its own packages.
2015-03-20 14:58:55,322 INFO  [ThreadPoolAsyncTaskExecutor::Thread 2]  c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:61886
2015-03-20 14:59:01,685 WARN  [spring-startup]  c.a.s.i.p.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin as it does not exist
2015-03-20 14:59:04,060 WARN  [spring-startup]  c.a.s.i.p.StaticPluginPathScanner Cannot scan directory /static/feature/tasks/icons/ in web static as it does not exist
2015-03-20 14:59:04,940 INFO  [spring-startup]  c.a.s.internal.ssh.server.SshServer Starting SSH server on port 7999...
2015-03-20 14:59:04,991 INFO  [spring-startup]  c.a.s.internal.ssh.server.SshServer Started SSH server successfully.
2015-03-20 14:59:05,033 INFO  [spring-startup]  c.a.s.internal.scm.PluginScmService Stash is using /usr/local/bin/git version 2.3.2

Can you check whether your STASH_HOME has been created properly? I'd expect a directory layout along the lines of:

bin
caches
export
lib
log
plugins
shared
tmp

If this doesn't exist, perhaps the user that's running Stash does not have write permissions on STASH_HOME?

Dillon Gilmore · Answer

Thanks for responding!

Here's what I am seeing there after shutting down stash and starting it back up then visiting it in atlassian-stash.log

[root@s1 log]# tail -f atlassian-stash.log
27-Mar-2015 10:08:17.883 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
27-Mar-2015 10:08:17.883 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-7990"]
27-Mar-2015 10:08:17.935 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
log4j:WARN No appenders could be found for logger (com.amazonaws.jmx.spi.SdkMBeanRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
27-Mar-2015 10:08:19.724 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-7990"]
27-Mar-2015 10:08:19.816 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-7990"]
27-Mar-2015 10:08:28.358 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
27-Mar-2015 10:08:28.360 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
27-Mar-2015 10:08:28.419 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /home/corp/atlassian/stash/3.7.0/lib/native:/home/corp/atlassian/application-data/stash/lib/native
27-Mar-2015 10:08:28.683 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-7990"]
27-Mar-2015 10:08:28.717 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
27-Mar-2015 10:08:28.721 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1112 ms
27-Mar-2015 10:08:28.758 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
27-Mar-2015 10:08:28.758 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.15
27-Mar-2015 10:08:44.064 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-7990"]
27-Mar-2015 10:08:44.081 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 15359 ms

Here's what I see on our iptables.

[root@s1 bin]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       icmp --  anywhere             anywhere            icmp timestamp-request 
DROP       icmp --  anywhere             anywhere            icmp timestamp-reply 
DROP       udp  --  anywhere             anywhere            udp dpt:snmp 
DROP       udp  --  anywhere             anywhere            udp dpt:snmptrap

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination