Port 7990 shuts down repeatedly

Question

I recently bought and installed Bitbucket server on an AWS RHEL 7.2. The install seemed to go well and the server comes up just fine. I also created an apache reverse proxy to be able to use a subdomain of bitbucket.company-domain.com. I've set up the server.xml file as follows; My apache virtualhost is set up as such; ServerName bitbucket.company-domain.com ProxyRequests Off ProxyTimeout 20 ProxyPreserveHost Off ProxyPass / http://localhost:7990/ ProxyPassReverse / http://localhost:7990/ Order Deny,Allow Allow from all Require all granted Require all granted CustomLog /var/log/httpd/bitbucket.company-domain.com-access.log combined ErrorLog /var/log/httpd/bitbucket.company-domain.com-error.log When I startup the Bitbucket server, netstat -tlpn shows this; Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 32495/mysqld tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1696/master tcp 0 0 0.0.0.0:22022 0.0.0.0:* LISTEN 20118/sshd tcp6 0 0 :::80 :::* LISTEN 5749/httpd tcp6 1 0 :::7990 :::* LISTEN 5951/java tcp6 0 0 127.0.0.1:7992 :::* LISTEN 5878/java tcp6 0 0 ::1:7992 :::* LISTEN 5878/java tcp6 0 0 ::1:25 :::* LISTEN 1696/master tcp6 0 0 :::22022 :::* LISTEN 20118/sshd Within less than 10 seconds though...port 7990 just vanishes and I get the Service Unavailable errors. Additionally, my apache error log file shows this for any requests; [Tue Jun 21 17:33:03.447264 2016] [proxy:error] [pid 5755] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:7990 (localhost) failed [Tue Jun 21 17:33:03.447309 2016] [proxy:error] [pid 5755] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s [Tue Jun 21 17:33:03.447320 2016] [proxy_http:error] [pid 5755] [client 65.29.83.112:63062] AH01114: HTTP: failed to make connection to backend: localhost [Tue Jun 21 17:33:03.575333 2016] [proxy:error] [pid 5752] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:7990 (localhost) failed [Tue Jun 21 17:33:03.575397 2016] [proxy:error] [pid 5752] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s [Tue Jun 21 17:33:03.575402 2016] [proxy_http:error] [pid 5752] [client 65.29.83.112:63063] AH01114: HTTP: failed to make connection to backend: localhost, referer: http://bitbucket.company-domain.com/ I have spent far too many hours trying to figure out where I've gone wrong. Can someone please help?

Jeff Thomas · Answer

Hi Rik,

A t2.micro instance does not have enough memory to run Bitbucket Server with Elasticsearch. I don't see any OutOfMemory errors in catalina.out, but I'd bet if we looked in the syslogs we'd see that the process was being killed due to lack of memory.

Please see our Supported Platforms page where we mention that a minimum of 3GB of memory is required on the server.

Rik Davis3 · Answer

The problem is resolved. Albeit, I have no clue what really fixed it.

I changed my instance on AWS from t2.micro to t2.small. I also rebooted the host as part of that effort. Somewhere in one of those two activities, it resolved the issue. I just wish I knew what it was so that I could know for future reference.

Rik Davis1 · Answer

Hi Jeff,

It does appear that catalina.out has some indications of issues. Maybe these are the cause?

21-Jun-2016 12:47:07.824 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/atlassian/bitbucket/4.7.1/lib/native:/var/atlassian/application-data/bitbucket/lib/native
21-Jun-2016 12:47:08.520 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-7990"]
21-Jun-2016 12:47:08.564 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
21-Jun-2016 12:47:08.575 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2063 ms
21-Jun-2016 12:47:08.633 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
21-Jun-2016 12:47:08.633 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.32
21-Jun-2016 12:47:24.199 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
21-Jun-2016 12:47:25.575 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-7990"]
21-Jun-2016 12:47:25.581 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 17006 ms
21-Jun-2016 13:20:14.533 WARNING [main] org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with directory [/var/atlassian/application-data/bitbucket/lib], exists: [false], isDirectory: [false], canRead: [false]
21-Jun-2016 13:20:14.539 WARNING [main] org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with directory [/var/atlassian/application-data/bitbucket/lib], exists: [false], isDirectory: [false], canRead: [false]
21-Jun-2016 13:20:15.846 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'redirectPort' to '' did not find a matching property.
21-Jun-2016 13:20:15.933 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.