Stash starts but hangs forever

Dillon Gilmore March 6, 2015

Hello,

I tested Stash 3.7.0 on my local machine and using the default install connecting to localhost:7990 works. Once I installed Stash 3.7.0 on a server and connected to my.domain:7990 it looks like it hangs on "connecting to my.domain:7990". I run stash in the foreground and here's what I see.

 

EDIT: On the same server I am successfully running an instance of JIRA, Confluence, and Crucible.

 

[root@s1 bin]# ./start-stash.sh -fg
Starting Atlassian Stash as current user

If you do not see a 'Server startup' message within 3 minutes, please see the troubleshooting guide at:

https://confluence.atlassian.com/display/STASHKB/Troubleshooting+Installation



-------------------------------------------------------------------------------
Stash is being run with a umask that contains potentially unsafe settings.
The following issues were found with the mask "u=rwx,g=rx,o=rx" (0022):
 - access is allowed to 'others'. It is recommended that 'others' be denied
   all access for security reasons.

The recommended umask for Stash is "u=,g=w,o=rwx" (0027) and can be
configured in setenv.sh
-------------------------------------------------------------------------------
Detecting JVM PermGen support...
PermGen switch is supported. Setting to 256m\n
Using STASH_HOME:      /home/corp/atlassian/application-data/stash
Using CATALINA_BASE:   /home/corp/atlassian/stash/3.7.0
Using CATALINA_HOME:   /home/corp/atlassian/stash/3.7.0
Using CATALINA_TMPDIR: /home/corp/atlassian/stash/3.7.0/temp
Using JRE_HOME:        /home/corp/atlassian-jira-4.3.4-standalone/jre
Using CLASSPATH:       /home/corp/atlassian/stash/3.7.0/bin/bootstrap.jar:/home/corp/atlassian/stash/3.7.0/bin/tomcat-juli.jar
Using CATALINA_PID:    /home/corp/atlassian/stash/3.7.0/work/catalina.pid
06-Mar-2015 11:05:47.018 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
06-Mar-2015 11:05:47.019 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
06-Mar-2015 11:05:47.068 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /home/corp/atlassian/stash/3.7.0/lib/native:/home/corp/atlassian/application-data/stash/lib/native
06-Mar-2015 11:05:47.234 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-7990"]
06-Mar-2015 11:05:47.260 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
06-Mar-2015 11:05:47.264 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 841 ms
06-Mar-2015 11:05:47.294 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
06-Mar-2015 11:05:47.294 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.15
06-Mar-2015 11:06:01.791 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-7990"]
06-Mar-2015 11:06:01.809 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 14545 ms

4 answers

0 votes
cirovladimir June 5, 2019

I'm sorry to bring back this old issue, but I just had the same problem described here, with no apparent error found in the logs nor elsewhere.

In our case it was due to a corrupted database caused by a server getting out of space and producing the error `FATAL: btree level 1 not found in index "19881"` which is [explained in more detail here](https://cirovladimir.wordpress.com/2018/12/04/postgres-btree-level-1-not-found-in-index/). After recovery with the pg_resetxlog some index errors were found and fixed by recreating the indexes with REINDEX (https://www.postgresql.org/docs/9.4/sql-reindex.html). We were able to restore the jira instance but the stash instance kept freezing at Initializing Spring context. I found that probably the cause would be a database lock which otherwise would be solved by issuing a `UPDATE databasechangeloglock SET LOCKED=false, LOCKGRANTED=null, LOCKEDBY=null where ID=1;` (https://stackoverflow.com/questions/54827560/bitbucket-migrating-home-directory-and-fails-to-start). I tried the aformentioned sql command but it got stuck, I even tried to kill sessions identified by `SELECT * FROM pg_stat_activity WHERE state = 'active';` and killed by `SELECT pg_terminate_backend(PID);`, restarted the database and reattempted the update but got stuck again. Now I was sure it was the culprit of stash not starting.

To solve this problem, we had to make a backup with pg_dumpall and then restore to a new instance (which will recreate all objects in the database and then import the data).

```
sudo su postgres
pg_dumpall > backup.dmp
exit
sudo service postgresql stop
mkdir -p /tmp/postgres
sudo chown postgres /tmp/postgres
sudo su postgres
cd /tmp/postgres
/usr/lib/postgresql/9.3/bin/initdb .
/usr/lib/postgresql/9.3/bin/postgres -D /tmp/postgres
```
in a new terminal restore the database
```
sudo su postgres
psql -f backup.dmp postgres
```
and finally restart jira and stash

```
sudo service jira start
sudo service atlstash start
```

 

Once restored, we're starting to upgrade our database and atlassian applications :P

0 votes
Michael Heemskerk
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 27, 2015

Hmmm... that does not look like it should. The output you've posted is what I would expect on the commandline where you started Stash (with the -fg option). The atlassian-stash.log file should contain something like: 2015-03-20 14:58:16,817 INFO [spring-startup] c.a.s.i.server.BuildInfoLogger Starting Stash 3.8.0-SNAPSHOT (${commi built on Fri Mar 20 14:58:16 CET 2015) 2015-03-20 14:58:16,822 INFO [spring-startup] c.a.s.i.server.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0-b132 2015-03-20 14:58:16,843 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /Users/mheemskerk/inst/clean 2015-03-20 14:58:18,865 WARN [spring-startup] com.jolbox.bonecp.BoneCPConfig JDBC password not set in driver properties, copying it from pool config 2015-03-20 14:58:20,892 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /Users/mheemskerk/inst/clean/shared 2015-03-20 14:58:45,988 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:61884 2015-03-20 14:58:52,675 WARN [spring-startup] c.a.p.o.f.t.s.ScanDescriptorForHostClassesStage The plugin 'atlassian-universal-plugin-manager-plugin-2.18.2.jar' uses a package 'org.apache.commons.fileupload.servlet' that is also exported by the application. It is highly recommended that the plugin use its own packages. 2015-03-20 14:58:55,322 INFO [ThreadPoolAsyncTaskExecutor::Thread 2] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:61886 2015-03-20 14:59:01,685 WARN [spring-startup] c.a.s.i.p.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin as it does not exist 2015-03-20 14:59:04,060 WARN [spring-startup] c.a.s.i.p.StaticPluginPathScanner Cannot scan directory /static/feature/tasks/icons/ in web static as it does not exist 2015-03-20 14:59:04,940 INFO [spring-startup] c.a.s.internal.ssh.server.SshServer Starting SSH server on port 7999... 2015-03-20 14:59:04,991 INFO [spring-startup] c.a.s.internal.ssh.server.SshServer Started SSH server successfully. 2015-03-20 14:59:05,033 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Stash is using /usr/local/bin/git version 2.3.2 Can you check whether your STASH_HOME has been created properly? I'd expect a directory layout along the lines of: bin caches export lib log plugins shared tmp If this doesn't exist, perhaps the user that's running Stash does not have write permissions on STASH_HOME?

0 votes
Dillon Gilmore March 27, 2015

Thanks for responding! Here's what I am seeing there after shutting down stash and starting it back up then visiting it in atlassian-stash.log [root@s1 log]# tail -f atlassian-stash.log 27-Mar-2015 10:08:17.883 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance. 27-Mar-2015 10:08:17.883 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-7990"] 27-Mar-2015 10:08:17.935 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina log4j:WARN No appenders could be found for logger (com.amazonaws.jmx.spi.SdkMBeanRegistry). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 27-Mar-2015 10:08:19.724 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-7990"] 27-Mar-2015 10:08:19.816 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-7990"] 27-Mar-2015 10:08:28.358 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. 27-Mar-2015 10:08:28.360 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. 27-Mar-2015 10:08:28.419 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /home/corp/atlassian/stash/3.7.0/lib/native:/home/corp/atlassian/application-data/stash/lib/native 27-Mar-2015 10:08:28.683 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-7990"] 27-Mar-2015 10:08:28.717 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 27-Mar-2015 10:08:28.721 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1112 ms 27-Mar-2015 10:08:28.758 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina 27-Mar-2015 10:08:28.758 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.15 27-Mar-2015 10:08:44.064 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-7990"] 27-Mar-2015 10:08:44.081 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 15359 ms Here's what I see on our iptables. [root@s1 bin]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination DROP icmp -- anywhere anywhere icmp timestamp-request DROP icmp -- anywhere anywhere icmp timestamp-reply DROP udp -- anywhere anywhere udp dpt:snmp DROP udp -- anywhere anywhere udp dpt:snmptrap Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination

0 votes
Michael Heemskerk
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 27, 2015

Do you see entries in the Stash logs under <STASH_HOME>/log/atlassian-stash.log and <STASH_HOME>/log/atlassian-stash-access.log? The atlassian-stash.log should at a minimum contain a blurb about Stash starting up. If you do see that Stash starts up correctly (in the log), but you don't see any entries in the access log, it could be that you have a firewall enabled on the server that's blocking port 7990?

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events