Bitbucket Elasticsearch Windows Service won't start after upgrading to 5.9.1

Question

After upgrading Bitbucket from version 5.5.0 to 5.9.1 I'm unable to get the elasticsearch windows service to start.  Starting it manually using the start-bitbucket.bat file works correctly but the windows service does not.  I've also tried switching the service user to atlbitbucket and granting owner rights to the Bitbucket installation directory and Bitbucket Home but still no success.

I noticed that the "config-template" folder needed to be updated to "config" in order for the service.bat install to run properly.  Maybe that has something to do with why the service will not start?

Any suggestions would be great.

Thank you,

jesse_oq35n6 · Answer

Anything standout after reviewing the logs?

Thank you,

jesse_oq35n6 · Answer

Hi Robhit,

Thank you for the reply.  Yes, I used the referenced instructions.  Which worked at 5.5.0 but doesn't appear to work with 5.9.1.  Seems like there are some additional steps in the service.bat for elastic search that we're there at previous versions.

Here is what I'm seeing in the search logs "bitbucket_search.log":

[2018-04-08T10:54:59,862][INFO ][o.e.n.Node ] [bitbucket_bundled] initializing ...
[2018-04-08T10:55:00,061][INFO ][o.e.e.NodeEnvironment ] [bitbucket_bundled] using [1] data paths, mounts [[DATAPART1 (D:)]], net usable_space [394.4gb], net total_space [847.9gb], spins? [unknown], types [NTFS]
[2018-04-08T10:55:00,062][INFO ][o.e.e.NodeEnvironment ] [bitbucket_bundled] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-04-08T10:55:00,123][INFO ][o.e.n.Node ] [bitbucket_bundled] node name [bitbucket_bundled], node ID [-BQiVWCFTaSCYLvNYk170g]
[2018-04-08T10:55:00,125][INFO ][o.e.n.Node ] [bitbucket_bundled] version[5.5.3], pid[3824], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Windows Server 2012/6.2/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-04-08T10:55:00,126][INFO ][o.e.n.Node ] [bitbucket_bundled] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss320k, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -Dpath.conf=D:\STASH_HOME\shared\search, -Delasticsearch, -Des.path.home=D:\Atlassian\Bitbucket\5.9.1\elasticsearch]
[2018-04-08T10:55:02,875][INFO ][c.a.e.b.c.BucklerConfig ] Basic authentication for HTTP transport enabled (auth.basic.http.enabled): true
[2018-04-08T10:55:02,877][INFO ][c.a.e.b.c.BucklerConfig ] Basic authentication for TCP transport enabled (auth.basic.tcp.enabled): true
[2018-04-08T10:55:02,881][INFO ][c.a.e.b.c.BucklerConfig ] TLS for HTTP transport enabled (tls.http.enabled): false
[2018-04-08T10:55:02,883][INFO ][c.a.e.b.c.BucklerConfig ] TLS for TCP transport enabled (tls.tcp.enabled): false
[2018-04-08T10:55:02,901][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [aggs-matrix-stats]
[2018-04-08T10:55:02,903][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [ingest-common]
[2018-04-08T10:55:02,905][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [lang-expression]
[2018-04-08T10:55:02,907][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [lang-groovy]
[2018-04-08T10:55:02,909][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [lang-mustache]
[2018-04-08T10:55:02,910][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [lang-painless]
[2018-04-08T10:55:02,912][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [parent-join]
[2018-04-08T10:55:02,913][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [percolator]
[2018-04-08T10:55:02,914][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [reindex]
[2018-04-08T10:55:02,916][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [transport-netty3]
[2018-04-08T10:55:02,918][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded module [transport-netty4]
[2018-04-08T10:55:02,921][INFO ][o.e.p.PluginsService ] [bitbucket_bundled] loaded plugin [buckler]
[2018-04-08T10:55:07,026][INFO ][o.e.d.DiscoveryModule ] [bitbucket_bundled] using discovery type [zen]
[2018-04-08T10:55:08,018][INFO ][o.e.n.Node ] [bitbucket_bundled] initialized
[2018-04-08T10:55:08,019][INFO ][o.e.n.Node ] [bitbucket_bundled] starting ...
[2018-04-08T10:55:09,453][INFO ][o.e.t.TransportService ] [bitbucket_bundled] publish_address {127.0.0.1:7993}, bound_addresses {127.0.0.1:7993}, {[::1]:7993}
[2018-04-08T10:55:12,577][INFO ][o.e.c.s.ClusterService ] [bitbucket_bundled] new_master {bitbucket_bundled}{-BQiVWCFTaSCYLvNYk170g}{rk40jxRBTY6mKKkupQyAAQ}{127.0.0.1}{127.0.0.1:7993}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-08T10:55:13,024][INFO ][o.e.g.GatewayService ] [bitbucket_bundled] recovered [1] indices into cluster_state
[2018-04-08T10:55:13,506][INFO ][c.a.e.b.SecureHttpServerTransport] [bitbucket_bundled] publish_address {127.0.0.1:7992}, bound_addresses {127.0.0.1:7992}, {[::1]:7992}
[2018-04-08T10:55:13,508][INFO ][o.e.n.Node ] [bitbucket_bundled] started
[2018-04-08T10:55:13,849][INFO ][o.e.c.r.a.AllocationService] [bitbucket_bundled] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[bitbucket-search-v1][1]] ...]).
[2018-04-08T10:56:37,505][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled] [bitbucket-search-v1/wmGQcFFrTzOZwbOIVTkBMA] deleting index
[2018-04-08T10:56:37,909][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled] [bitbucket-search-v1] creating index, cause [api], templates [], shards [5]/[1], mappings [file, project, repository, indexstate]
[2018-04-08T10:56:38,213][INFO ][o.e.c.m.MetaDataMappingService] [bitbucket_bundled] [bitbucket-search-v1/sR4-hnnSTlORxydvF8NfJw] create_mapping [indexversion]
[2018-04-08T10:58:54,783][INFO ][o.e.b.Bootstrap ] running graceful exit on windows
[2018-04-08T10:58:54,786][INFO ][o.e.n.Node ] [bitbucket_bundled] stopping ...
[2018-04-08T10:58:55,012][INFO ][o.e.n.Node ] [bitbucket_bundled] stopped
[2018-04-08T10:58:55,013][INFO ][o.e.n.Node ] [bitbucket_bundled] closing ...
[2018-04-08T10:58:55,033][INFO ][o.e.n.Node ] [bitbucket_bundled] closed

"bitbucket_search_deprecation.log"

[2018-04-07T11:28:19,891][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [D:\STASH_HOME\shared\search\data], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path
[2018-04-08T10:54:59,919][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [D:\STASH_HOME\shared\search\data], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path
[2018-04-08T10:56:38,204][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [version]
[2018-04-08T10:56:38,208][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [version]

Let me know if you need any additional information.

Thank you,

robhit · Answer

Hi Jesse,

Thanks for your question!

I would ask what do you see in the service logs when starting Bitbucket? That might give us some hints on the issue.

Hope you are following the below page:

* Running-bitbucket-server-as-a-windows-service

Thanks,

Robhit