Search is currently Unavailable :: failed to obtain node locks (Window Server)

[Richard Presley]

So I've been trying to get the Search functionality working in our bitbucket server for a time now.

Every time we try to use the search bar we get a "Search is currently Unavailable" message or gives us a black screen.

Checking the logs I get this error:

[2018-09-18T14:37:50,315][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\Atlassian\ApplicationData\Bitbucket\shared\search\data\bitbucket_search]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.node.Node.<init>(Node.java:244) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.5.3.jar:5.5.3]
[2018-09-18T14:37:50,330][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [bitbucket_bundled] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\Atlassian\ApplicationData\Bitbucket\shared\search\data\bitbucket_search]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.5.3.jar:5.5.3]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\Atlassian\ApplicationData\Bitbucket\shared\search\data\bitbucket_search]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.node.Node.<init>(Node.java:244) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.5.3.jar:5.5.3]
... 6 more

now this error is not new. I've been following the number of solutions for this common problem, Like:

• Stopping and starting the ElasticSearch service(command prompt and Task Manager)
• Restarting Bitbucket
• verifying there is only one node[0]

but none of these solves the problem.

Update

So I restarted Bitbucket and received another error that may help narrow down the issue

2018-09-21 11:02:03,495 DEBUG [http-nio-80-exec-7] @19013IJx662x40x0 1ti1vm9 127.0.0.1 "PUT /mvc/error401 HTTP/1.1" c.a.s.i.web.ErrorPageController User is not authorized to access [/bitbucket-search-v1]
2018-09-21 11:02:03,541 ERROR [Caesium-1-1] c.a.b.s.i.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch
2018-09-21 11:02:03,541 WARN [Caesium-1-1] c.a.b.s.i.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.search.internal.indexing.jobs.StartupChecksJob Job id: 36874242-b6f7-4b4a-841a-4bec40e13359
2018-09-21 11:02:03,541 DEBUG [Caesium-1-1] c.a.b.s.i.c.ClusterJobScheduler Rescheduling failed job. Current retry: 3

[Christian Glockner]

Hi Richard,

A quick Google search suggests that this can happen if there's already a java process belonging to Elasticsearch running. After stopping Elasticsearch, do you still see a java process belonging to it in Task Manager?

Cheers,

Christian

Premier Support Engineer

Atlassian

[Richard Presley]

After stopping the bitbucket server elasticsearch service, the following java process goes away.