nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

elasticsearch not running with bitbucket v6.10

Antonio Medač_gvo67p

bitbucket version: 6.10.0.52

As seen from logs and troubleshooting, elasticsearch is not running:

[root@bitbucket ~]# ps -ef |grep [e]lastic 
[root@bitbucket ~]# tail /data/bitbucket/log/search/bitbucket_search.log 
[2020-02-05T16:17:05,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:17:35,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:05,813][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:35,814][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:05,815][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:35,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:58,488][INFO ][o.e.n.Node ] [bitbucket_bundled]stopping ...
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]stopped
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]closing ...
[2020-02-05T16:20:01,741][INFO ][o.e.n.Node ] [bitbucket_bundled]closed
[root@bitbucket ~]# telnet localhost 7992
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

from /data/bitbucket/log/atlassian-bitbucket.log:

2020-02-07 06:14:20,905 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation. 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: e593ff34-b85c-4fb7-ab94-faca3409cfe4

Also test from website is not working, see attached picture:

I tried curl commands, tried to reset password but can't do any of it since elasticsearch is not started.

What else could I try?

Best Regards, Antonio

Accepted answers

Antonio Medač_gvo67p

SOLVED: There was two issues:

1. wrong (root) permissions on elasticsearch folder: /opt/bitbucket/elasticsearch/

2. Broken index, solved by folowing rebuild index page.

All comments

Hitendra Chauhan

This problem seems to be due to low disk space. Try increasing the disk space of the mount point on which data folder is located.

Antonio Medač_gvo67p

Hi,

thank you for response and help, but it is not disk space issue, it was just temporary there....

[root@bitbucket buckler]# df -h /data/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 15G 5.3G 74% /

As mentioned, connection doesn't work from web interface like I posted in screenshot above. It is not "refused", just failed to establish connection (which is normal because elastic search is not running).

I tried to reset password but it doesn't help (retreived values from $BITBUCKET_HOME/shared/search/buckler/buckler.yml file)

Here is what is shown in log, service status and evidence there is no two processes running as it should when you grep for bitbucket:

[root@bitbucket buckler]# ps -ef |grep [b]itb
tdsuser 15709 1 18 06:56 ? 00:16:15 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/data/bitbucket/tmp -Djava.library.path=/opt/bitbucket/lib/native;/data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start
[root@bitbucket buckler]# grep -i elasticsearch /var/log/bitbucket/atlassian-bitbucket.log
2020-02-10 06:57:30,689 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2020-02-10 06:57:45,800 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 06:58:45,820 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:00:45,834 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:04:45,849 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:12:45,980 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:22:45,993 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:32:46,006 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:42:46,019 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:52:46,038 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:02:46,052 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:12:46,066 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
[root@bitbucket buckler]# systemctl status bitbucket
● bitbucket.service - Atlassian Bitbucket Server Service
Loaded: loaded (/usr/lib/systemd/system/bitbucket.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-10 06:56:15 UTC; 1h 25min ago
Process: 15562 ExecStop=/opt/bitbucket/bin/stop-bitbucket.sh (code=exited, status=0/SUCCESS)
Process: 15622 ExecStart=/opt/bitbucket/bin/start-bitbucket.sh (code=exited, status=0/SUCCESS)
Main PID: 15709 (java)
CGroup: /system.slice/bitbucket.service
└─15709 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m...
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: - Access is allowed to 'others'. It is recommended that 'others' be denied
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: all access for security reasons.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: configured in _start-webapp.sh
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The current open files limit is set to less than 4096
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Attempting to increase limit...
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Limit increased to 4096 open files
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Starting Bitbucket webapp at http://localhost:7990
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The Bitbucket webapp has been started.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com systemd[1]: Started Atlassian Bitbucket Server Service.

The main problem here is how to start elasticsearch?

Best Regards, //Antonio

Antonio Medač_gvo67p

SOLVED: There was two issues:

1. wrong (root) permissions on elasticsearch folder: /opt/bitbucket/elasticsearch/

2. Broken index, solved by folowing rebuild index page.

Kaspars Laizans

When running 7.16 via systemd, I had an issue with missing JAVA_HOME env var. Then it would try to use builtin JRE, which was not present and fail. Only log showing this was journalctl service log, in atlassian_bitbucket.log and search/bitbucket_search.log nothing would be visible.

Added "Environment=JAVA_HOME=/path/to/java" in service definition and it managed to start up bundled elasticsearch instance

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

server

bitbucket

bitbucket-server

elastic-search