Elasticsearch fata error after upgrade 5.3.1 => 6.2.0

[Lawrence Sampas]

After upgrading from 5.3.1 to 6.2.0, Elasticsearch won't start.  Elasticsearch logs show a fatal error. I've checked java home, home dir, paths, but I can't figure out this one yet.

[2019-06-17T09:25:24,394][INFO ][c.a.e.b.c.BucklerConfig ] [bitbucket_bundled]Basic authentication for HTTP transport enabled (auth.basic.http.enabled): true
[2019-06-17T09:25:24,394][INFO ][c.a.e.b.c.BucklerConfig ] [bitbucket_bundled]Basic authentication for TCP transport enabled (auth.basic.tcp.enabled): true
[2019-06-17T09:25:24,396][INFO ][c.a.e.b.c.BucklerConfig ] [bitbucket_bundled]TLS for HTTP transport enabled (tls.http.enabled): false
[2019-06-17T09:25:24,397][INFO ][c.a.e.b.c.BucklerConfig ] [bitbucket_bundled]TLS for TCP transport enabled (tls.tcp.enabled): false
[2019-06-17T09:25:24,402][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [aggs-matrix-stats]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [analysis-common]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [ingest-common]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [lang-expression]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [lang-mustache]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [lang-painless]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [mapper-extras]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [parent-join]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [percolator]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [rank-eval]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [reindex]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [repository-url]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [transport-netty4]
[2019-06-17T09:25:24,403][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded module [tribe]
[2019-06-17T09:25:24,404][INFO ][o.e.p.PluginsService ] [bitbucket_bundled]loaded plugin [buckler]

[2019-06-17T09:25:24,938][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [bitbucket_bundled]fatal error in thread [main], exiting
java.lang.ExceptionInInitializerError: null
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_211]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_211]
at org.elasticsearch.painless.lookup.PainlessLookupBuilder.addPainlessClass(PainlessLookupBuilder.java:176) ~[?:?]
at org.elasticsearch.painless.lookup.PainlessLookupBuilder.buildFromWhitelists(PainlessLookupBuilder.java:70) ~[?:?]
at org.elasticsearch.painless.PainlessScriptEngine.<init>(PainlessScriptEngine.java:109) ~[?:?]
at org.elasticsearch.painless.PainlessPlugin.getScriptEngine(PainlessPlugin.java:89) ~[?:?]
at org.elasticsearch.script.ScriptModule.<init>(ScriptModule.java:80) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.node.Node.<init>(Node.java:362) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.3.jar:6.5.3]

[Lawrence Sampas]

And Japanese Era is there:

Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
at java.time.chrono.JapaneseEra.<clinit>(JapaneseEra.java:179) ~[?:1.8.0_211]
... 19 more

(oh, and we're on Ubuntu Ubuntu 16.04.6 LTS with 5.7.26-0ubuntu0.16.04.1.

java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

[Daniel Santos]

I would try to install a newer build of Java 1.8 as recommended in the post I shared. A version that is equal or above 1.8.0_212.

[Daniel Santos]

Hi @Lawrence Sampas

I'm not experienced in Bitbucket server but I wanted to share my 5 cents on this one.
I found this article with pretty similar error lines:

• Error at java.time.chrono.JapaneseEra - Elasticsearch - Discuss the Elastic Stack

Can you please check if you have a caused by section after the error you shared? If you have please share it here as well.
Please also check if the Java version used is supported as per: 

• Supported platforms - Atlassian Documentation

I'll wait for your findings.