Can't get bitbucket running after upgrade

[Justin Campo]

Version 4.8.3 was running perfectly.  I upgraded to 5.5.2 and I can't get it work any longer.  I used the install/upgrade script to install.  Everything is default, I have no plugins, I am running apache as a proxy so I can use url with ssl.

So on a centos 6 server....

/var/atassian/application-data/bitbucket/log/atlassian-bitbucket.log

------------------------------------------------------------------------------

2017-12-29 17:40:54,379 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.6.RELEASE)

2017-12-29 17:40:54,778 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on base.questionhosting.com with PID 33350 (/opt/atlassian/bitbucket/5.5.2/app/WEB-INF/classes started by atlbitbucket in /opt/atlassian/bitbucket/5.5.2/bin)
2017-12-29 17:40:54,778 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2017-12-29 17:40:56,377 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.5.2 (f0a715d built on Sun Nov 26 21:21:53 EST 2017)
2017-12-29 17:40:56,377 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_151-b12
2017-12-29 17:40:58,787 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 6.597 seconds (JVM running for 9.182)
2017-12-29 17:41:03,253 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2017-12-29 17:41:10,375 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2017-12-29 17:41:13,215 WARN [spring-startup] com.hazelcast.instance.Node [199.101.185.186]:5701 [atlbitbucket] [3.7.4-atlassian-43] No join method is enabled! Starting standalone.
2017-12-29 17:41:42,574 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:22976
2017-12-29 17:42:13,230 WARN [ThreadPoolAsyncTaskExecutor::Thread 5] c.a.p.s.s.util.ProductFilterUtil Couldn't detect product, will use ProductFilter.ALL
2017-12-29 17:42:14,736 WARN [ThreadPoolAsyncTaskExecutor::Thread 16] c.a.p.s.s.util.ProductFilterUtil Couldn't detect product, will use ProductFilter.ALL
2017-12-29 17:42:14,820 WARN [ThreadPoolAsyncTaskExecutor::Thread 6] c.a.p.s.s.util.ProductFilterUtil Couldn't detect product, will use ProductFilter.ALL
2017-12-29 17:42:16,383 INFO [ThreadPoolAsyncTaskExecutor::Thread 12] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:19190
2017-12-29 17:42:29,542 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2017-12-29 17:42:32,454 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2017-12-29 17:42:32,511 INFO [spring-startup] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2017-12-29 17:42:32,528 INFO [spring-startup] c.a.b.s.i.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2017-12-29 17:42:32,529 INFO [spring-startup] c.a.b.s.j.c.i.HealthCheckRunner New health check registered: AuthenticationCheck
2017-12-29 17:42:32,649 INFO [spring-startup] c.a.b.s.i.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2017-12-29 17:42:32,650 INFO [spring-startup] c.a.b.s.j.c.i.HealthCheckRunner New health check registered: KnownIssueCheck
2017-12-29 17:42:32,657 INFO [spring-startup] c.a.b.s.j.c.i.HealthCheckRunner New health check registered: SearchIndexCheck
2017-12-29 17:42:32,657 INFO [spring-startup] c.a.b.s.i.c.DefaultClusterJobManager Registering job for StartupChecksJob
2017-12-29 17:42:33,835 WARN [Caesium-1-4] c.a.a.c.DefaultApplinkStatusService Unrecognized error while attempting to retrieve status of Application Link '38791d59-118f-39e7-914f-d903b62f89c1'
2017-12-29 17:42:33,841 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/bin/git version 2.8.0
2017-12-29 17:42:34,311 INFO [spring-startup] c.a.s.i.hazelcast.HazelcastLifecycle Updating Hazelcast instance capabilities
2017-12-29 17:42:36,885 WARN [UpmScheduler:thread-1] com.atlassian.upm.pac.PacClientImpl Update check request may take longer because of the number of add-ons
2017-12-29 17:42:42,398 INFO [Caesium-1-4] c.a.b.s.j.c.i.HealthCheckRunner Running up to 3 search health checks
2017-12-29 17:42:42,496 INFO [pool-20-thread-1] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed due to exception java.net.ConnectException: Connection refused
2017-12-29 17:42:42,639 INFO [Caesium-1-4] c.a.b.s.j.c.i.HealthCheckRunner Health checks will run again after PT5M
2017-12-29 17:42:42,743 INFO [Caesium-1-2] c.a.b.s.i.i.jobs.StartupChecksJob Attempting provisioning in startup job
2017-12-29 17:42:42,764 INFO [pool-21-thread-1] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed due to exception java.net.ConnectException: Connection refused

--------------------------------------

Can you please help, I have no idea what is wrong.

[edwin]

Hi Justin,

What error is showing on the browser?

[Justin Campo]

Service Temporarily Unavailable

apache ssl_error_log says the following 

---------------------------------------

[Sat Dec 30 00:42:59 2017] [error] (111)Connection refused: proxy: AJP: attempt to connect to 127.0.0.1:7991 (localhost) failed
[Sat Dec 30 00:42:59 2017] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Sat Dec 30 00:42:59 2017] [error] proxy: AJP: failed to make connection to backend: localhost

/etc/httpd/conf.d/ssl.conf

------------------

<Proxy *>
Order deny,allow
Allow from all
</Proxy>

ProxyPass /repo ajp://localhost:7991/repo
ProxyPassReverse /repo ajp://localhost:7991/repo
<Location /repo>
Order allow,deny
Allow from all
</Location>

/server.xml

----------------

<Server port="8006" shutdown="SHUTDOWN">
<!-- Security listener. Documentation at /docs/config/listeners.html
<Listener className="org.apache.catalina.security.SecurityListener" />
-->
<!--APR library loader. Documentation at /docs/apr.html -->
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<!-- Prevent memory leaks due to use of particular java/javax APIs-->
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<!--Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /-->

<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>
<!-- Editable user database that can also be used by
UserDatabaseRealm to authenticate users
-->

<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>

<!-- A "Service" is a collection of one or more "Connectors" that share
a single "Container" Note: A "Service" is not itself a "Container",
so you may not define subcomponents such as "Valves" at this level.
Documentation at /docs/config/service.html
-->
<Service name="Catalina">

<!--The connectors can use a shared executor, you can define one or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
-->

<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL HTTP/1.1 Connector on port 7990

If you change this port, you have to update scripts.cfg as well
to have the same port configuration.
-->
<Connector port="7989" protocol="HTTP/1.1"
connectionTimeout="20000"

useBodyEncodingForURI="true"
redirectPort="8443"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript" />

<Connector port="7990" protocol="AJP/1.3"
connectionTimeout="20000"
useBodyEncodingForURI="true"
redirectPort="8443"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript" />

<Connector port="7991" protocol="AJP/1.3"
connectionTimeout="20000"
useBodyEncodingForURI="true"
redirectPort="8443"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript"
scheme="https"
proxyName="questionadmin.com"
proxyPort="443"
secure="true"/>

<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="7990" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
-->
<!-- Define a SSL HTTP/1.1 Connector on port 8443
This connector uses the NIO implementation that requires the JSSE
style configuration. When using the APR/native implementation, the
OpenSSL style configuration is required as described in the APR/native

<!-- An Engine represents the entry point (within Catalina) that processes
every request. The Engine implementation for Tomcat stand alone
analyzes the HTTP headers included with the request, and passes them
on to the appropriate Host (virtual host).
Documentation at /docs/config/engine.html -->

<!-- You should set jvmRoute to support load-balancing via AJP ie :
<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
-->
<Engine name="Catalina" defaultHost="localhost">

<!-- For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->

<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
<!-- This Realm uses the UserDatabase configured in the global JNDI
resources under the key "UserDatabase". Any edits
that are performed against this UserDatabase are immediately
available for use by the Realm. -->
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>

<Host name="localhost" appBase="webapps"

unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">

<!-- Before changing the context path for Bitbucket Server, please read our documentation:
https://confluence.atlassian.com/x/5oCNEQ

You will have to update scripts.cfg
AND rename webapps/ROOT directory. -->

<Context docBase="${catalina.home}/atlassian-bitbucket"
path="/repo"
reloadable="false"
useHttpOnly="true"/>

<!-- SingleSignOn valve, share authentication between web applications
Documentation at: /docs/config/valve.html -->
<!--
<Valve className="org.apache.catalina.authenticator.SingleSignOn" />
-->

<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<!--
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t &quot;%r&quot; %s %b" />
-->
</Host>
</Engine>
</Service>
</Server>

[Justin Campo]

So when I start bitbucket I don't get any errors....its only every 10 minutes it seems like their is a problem connecting to elasticsearch.

I think the problem is that I didn't realize you guys phased out server.xml (mine is pasted above).

I converted it to the following bitbucket.properties:

jdbc.driver=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://127.0.0.1:3306/jirabitbucket_db?characterEncoding=utf8&useUnicode=true&sessionVariables=storage_engine%3DInnoDB
jdbc.user=xxxxxxx
jdbc.password=xxxx

server.secure=true
server.scheme=https
server.port=7989
server.redirect-port=8443
server.require-ssl=true

server.additional-connector.1.port=7990
server.additional-connector.1.redirect-port=8443
server.additional-connector.1.require-ssl=true

server.additional-connector.2.port=7991
server.additional-connector.2.redirect-port=8443
server.additional-connector.2.require-ssl=true

But I its not working...

apaches error log

----------------------

Dec 30 14:16:10 2017] [error] ajp_read_header: ajp_ilink_receive failed
[Sat Dec 30 14:16:10 2017] [error] (120007)APR does not understand this error code: proxy: read response failed from (null) (localhost)
[Sat Dec 30 14:16:14 2017] [error] ajp_read_header: ajp_ilink_receive failed
[Sat Dec 30 14:16:14 2017] [error] (120007)APR does not understand this error code: proxy: read response failed from [::1]:7991 (localhost)
[Sat Dec 30 14:16:32 2017] [error] ajp_read_header: ajp_ilink_receive failed
[Sat Dec 30 14:16:32 2017] [error] (120007)APR does not understand this error code: proxy: read response failed from [::1]:7991 (localhost)
[Sat Dec 30 14:16:33 2017] [error] ajp_read_header: ajp_ilink_receive failed
[Sat Dec 30 14:16:33 2017] [error] (120007)APR does not understand this error code: proxy: read response failed from [::1]:7991 (localhost)
[Sat Dec 30 14:16:42 2017] [error] ajp_read_header: ajp_ilink_receive failed
[Sat Dec 30 14:16:42 2017] [error] (120007)APR does not understand this error code: proxy: read response failed from [::1]:7991 (localhost)

Any ideas...I've put 10+ hours into this I am getting no where...this upgrade has been really frustrating.

[Justin Campo]

So it looks like you stopped supporting AJP.  I assume there is no way to accomplish what I am looking for? Can I at least get a refund for the money I spent on your system that I can no longer use?