Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,361,143
Community Members
 
Community Events
168
Community Groups

elasticsearch not running with bitbucket v6.10

bitbucket version: 6.10.0.52

As seen from logs and troubleshooting, elasticsearch is not running:

[root@bitbucket ~]# ps -ef |grep [e]lastic 
[root@bitbucket ~]# tail /data/bitbucket/log/search/bitbucket_search.log
[2020-02-05T16:17:05,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:17:35,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:05,813][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:35,814][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:05,815][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:35,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:58,488][INFO ][o.e.n.Node ] [bitbucket_bundled]stopping ...
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]stopped
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]closing ...
[2020-02-05T16:20:01,741][INFO ][o.e.n.Node ] [bitbucket_bundled]closed
[root@bitbucket ~]# telnet localhost 7992
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

from /data/bitbucket/log/atlassian-bitbucket.log:

2020-02-07 06:14:20,905 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation. 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: e593ff34-b85c-4fb7-ab94-faca3409cfe4

 Also test from website is not working, see attached picture:

image-2020-02-07-07-13-23-813.png

I tried curl commands, tried to reset password but can't do any of it since elasticsearch is not started.

What else could I try?

Best Regards, Antonio

3 answers

1 accepted

0 votes
Answer accepted

SOLVED: There was two issues:

1. wrong (root) permissions on elasticsearch folder: /opt/bitbucket/elasticsearch/

2. Broken index, solved by folowing rebuild index page.

When running 7.16 via systemd, I had an issue with missing JAVA_HOME env var. Then it would try to use builtin JRE, which was not present and fail. Only log showing this was journalctl service log, in atlassian_bitbucket.log and search/bitbucket_search.log nothing would be visible.

Added "Environment=JAVA_HOME=/path/to/java" in service definition and it managed to start up bundled elasticsearch instance

This problem seems to be due to low disk space. Try increasing the disk space of the mount point on which data folder is located.

Hi,

thank you for response and help, but it is not disk space issue, it was just temporary there.... :(

[root@bitbucket buckler]# df -h /data/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 15G 5.3G 74% /

As mentioned, connection doesn't work from web interface like I posted in screenshot above. It is not "refused", just failed to establish connection (which is normal because elastic search is not running).

I tried to reset password but it doesn't help (retreived values from $BITBUCKET_HOME/shared/search/buckler/buckler.yml file)

Here is what is shown in log, service status and evidence there is no two processes running as it should when you grep for bitbucket:

 

[root@bitbucket buckler]# ps -ef |grep [b]itb
tdsuser 15709 1 18 06:56 ? 00:16:15 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/data/bitbucket/tmp -Djava.library.path=/opt/bitbucket/lib/native;/data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start

[root@bitbucket buckler]# grep -i elasticsearch /var/log/bitbucket/atlassian-bitbucket.log
2020-02-10 06:57:30,689 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2020-02-10 06:57:45,800 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 06:58:45,820 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:00:45,834 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:04:45,849 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:12:45,980 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:22:45,993 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:32:46,006 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:42:46,019 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:52:46,038 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:02:46,052 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:12:46,066 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
[root@bitbucket buckler]# systemctl status bitbucket
● bitbucket.service - Atlassian Bitbucket Server Service
Loaded: loaded (/usr/lib/systemd/system/bitbucket.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-10 06:56:15 UTC; 1h 25min ago
Process: 15562 ExecStop=/opt/bitbucket/bin/stop-bitbucket.sh (code=exited, status=0/SUCCESS)
Process: 15622 ExecStart=/opt/bitbucket/bin/start-bitbucket.sh (code=exited, status=0/SUCCESS)
Main PID: 15709 (java)
CGroup: /system.slice/bitbucket.service
└─15709 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m...

Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: - Access is allowed to 'others'. It is recommended that 'others' be denied
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: all access for security reasons.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: configured in _start-webapp.sh
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The current open files limit is set to less than 4096
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Attempting to increase limit...
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Limit increased to 4096 open files
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Starting Bitbucket webapp at http://localhost:7990
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The Bitbucket webapp has been started.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com systemd[1]: Started Atlassian Bitbucket Server Service.

 

The main problem here is how to start elasticsearch?

Best Regards, //Antonio

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Git push size limits are coming to Bitbucket Cloud starting April 4th, 2022

Beginning on April 4th, we will be implementing push limits. This means that your push cannot be completed if it is over 3.5 GB. If you do attempt to complete a push that is over 3.5 GB, it will fail...

3,487 views 3 10
Read article

Atlassian Community Events