Port 7990 shuts down repeatedly

[Rik Davis]

I recently bought and installed Bitbucket server on an AWS RHEL 7.2.

The install seemed to go well and the server comes up just fine. I also created an apache reverse proxy to be able to use a subdomain of bitbucket.company-domain.com.

I've set up the server.xml file as follows;

<Connector port="7990" protocol="HTTP/1.1"
                   connectionTimeout="20000"
                   useBodyEncodingForURI="true"
                   redirectPort=""
                   compression="on"
                   compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript"
                   scheme="http"
                   proxyName="bitbucket.company-domain.com"
                   proxyPort="80" />

My apache virtualhost is set up as such;

<VirtualHost *:80>
  ServerName bitbucket.company-domain.com
  ProxyRequests Off
  ProxyTimeout 20
  ProxyPreserveHost Off
  ProxyPass               /       http://localhost:7990/
  ProxyPassReverse        /       http://localhost:7990/
  <Proxy *>
    Order Deny,Allow
    Allow from all
    Require all granted
  </Proxy>
  <Location />
    Require all granted
  </Location>
  CustomLog /var/log/httpd/bitbucket.company-domain.com-access.log combined
  ErrorLog /var/log/httpd/bitbucket.company-domain.com-error.log
</VirtualHost>

When I startup the Bitbucket server, netstat -tlpn shows this;

Proto Recv-Q Send-Q Local Address    Foreign Address       State       PID/Program name
tcp        0      0 0.0.0.0:3306     0.0.0.0:*             LISTEN      32495/mysqld
tcp        0      0 127.0.0.1:25     0.0.0.0:*               LISTEN      1696/master
tcp        0      0 0.0.0.0:22022    0.0.0.0:*               LISTEN      20118/sshd
tcp6       0      0 :::80            :::*                    LISTEN      5749/httpd
tcp6       1      0 :::7990          :::*                    LISTEN      5951/java
tcp6       0      0 127.0.0.1:7992   :::*                    LISTEN      5878/java
tcp6       0      0 ::1:7992         :::*                    LISTEN      5878/java
tcp6       0      0 ::1:25           :::*                    LISTEN      1696/master
tcp6       0      0 :::22022         :::*                    LISTEN      20118/sshd

Within less than 10 seconds though...port 7990 just vanishes and I get the Service Unavailable errors.

Additionally, my apache error log file shows this for any requests;

[Tue Jun 21 17:33:03.447264 2016] [proxy:error] [pid 5755] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:7990 (localhost) failed
[Tue Jun 21 17:33:03.447309 2016] [proxy:error] [pid 5755] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Tue Jun 21 17:33:03.447320 2016] [proxy_http:error] [pid 5755] [client 65.29.83.112:63062] AH01114: HTTP: failed to make connection to backend: localhost
[Tue Jun 21 17:33:03.575333 2016] [proxy:error] [pid 5752] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:7990 (localhost) failed
[Tue Jun 21 17:33:03.575397 2016] [proxy:error] [pid 5752] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Tue Jun 21 17:33:03.575402 2016] [proxy_http:error] [pid 5752] [client 65.29.83.112:63063] AH01114: HTTP: failed to make connection to backend: localhost, referer: http://bitbucket.company-domain.com/

I have spent far too many hours trying to figure out where I've gone wrong. Can someone please help?

[Jeff Thomas]

Hi Rik,

A t2.micro instance does not have enough memory to run Bitbucket Server with Elasticsearch. I don't see any OutOfMemory errors in catalina.out, but I'd bet if we looked in the syslogs we'd see that the process was being killed due to lack of memory.

Please see our Supported Platforms page where we mention that a minimum of 3GB of memory is required on the server.

[Rik Davis]

The problem is resolved. Albeit, I have no clue what really fixed it.

I changed my instance on AWS from t2.micro to t2.small. I also rebooted the host as part of that effort. Somewhere in one of those two activities, it resolved the issue. I just wish I knew what it was so that I could know for future reference.

[Rik Davis]

Hi Jeff,

It does appear that catalina.out has some indications of issues. Maybe these are the cause?

21-Jun-2016 12:47:07.824 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/atlassian/bitbucket/4.7.1/lib/native:/var/atlassian/application-data/bitbucket/lib/native
21-Jun-2016 12:47:08.520 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-7990"]
21-Jun-2016 12:47:08.564 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
21-Jun-2016 12:47:08.575 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2063 ms
21-Jun-2016 12:47:08.633 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
21-Jun-2016 12:47:08.633 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.32
21-Jun-2016 12:47:24.199 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
21-Jun-2016 12:47:25.575 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-7990"]
21-Jun-2016 12:47:25.581 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 17006 ms
21-Jun-2016 13:20:14.533 WARNING [main] org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with directory [/var/atlassian/application-data/bitbucket/lib], exists: [false], isDirectory: [false], canRead: [false]
21-Jun-2016 13:20:14.539 WARNING [main] org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with directory [/var/atlassian/application-data/bitbucket/lib], exists: [false], isDirectory: [false], canRead: [false]
21-Jun-2016 13:20:15.846 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'redirectPort' to '' did not find a matching property.
21-Jun-2016 13:20:15.933 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.

[Rik Davis]

I'm assuming you mean the instance type?

If so, then that would be the t2.micro

[Jeff Thomas]

There should be some additional information in the following log files which indicates why Bitbucket Server is stopping:

- $BITBUCKET_HOME/log/atlassian-bitbucket.log
- <Bitbucket installation directory>/logs/catalina.out

Which type of EC2 instance are you using?