Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

elasticsearch not running with bitbucket v6.10

bitbucket version: 6.10.0.52

As seen from logs and troubleshooting, elasticsearch is not running:

[root@bitbucket ~]# ps -ef |grep [e]lastic 
[root@bitbucket ~]# tail /data/bitbucket/log/search/bitbucket_search.log
[2020-02-05T16:17:05,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:17:35,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:05,813][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:35,814][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:05,815][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:35,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:58,488][INFO ][o.e.n.Node ] [bitbucket_bundled]stopping ...
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]stopped
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]closing ...
[2020-02-05T16:20:01,741][INFO ][o.e.n.Node ] [bitbucket_bundled]closed
[root@bitbucket ~]# telnet localhost 7992
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

from /data/bitbucket/log/atlassian-bitbucket.log:

2020-02-07 06:14:20,905 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation. 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: e593ff34-b85c-4fb7-ab94-faca3409cfe4

 Also test from website is not working, see attached picture:

image-2020-02-07-07-13-23-813.png

I tried curl commands, tried to reset password but can't do any of it since elasticsearch is not started.

What else could I try?

Best Regards, Antonio

3 answers

1 accepted

0 votes
Answer accepted

SOLVED: There was two issues:

1. wrong (root) permissions on elasticsearch folder: /opt/bitbucket/elasticsearch/

2. Broken index, solved by folowing rebuild index page.

When running 7.16 via systemd, I had an issue with missing JAVA_HOME env var. Then it would try to use builtin JRE, which was not present and fail. Only log showing this was journalctl service log, in atlassian_bitbucket.log and search/bitbucket_search.log nothing would be visible.

Added "Environment=JAVA_HOME=/path/to/java" in service definition and it managed to start up bundled elasticsearch instance

This problem seems to be due to low disk space. Try increasing the disk space of the mount point on which data folder is located.

Hi,

thank you for response and help, but it is not disk space issue, it was just temporary there.... :(

[root@bitbucket buckler]# df -h /data/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 15G 5.3G 74% /

As mentioned, connection doesn't work from web interface like I posted in screenshot above. It is not "refused", just failed to establish connection (which is normal because elastic search is not running).

I tried to reset password but it doesn't help (retreived values from $BITBUCKET_HOME/shared/search/buckler/buckler.yml file)

Here is what is shown in log, service status and evidence there is no two processes running as it should when you grep for bitbucket:

 

[root@bitbucket buckler]# ps -ef |grep [b]itb
tdsuser 15709 1 18 06:56 ? 00:16:15 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/data/bitbucket/tmp -Djava.library.path=/opt/bitbucket/lib/native;/data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start

[root@bitbucket buckler]# grep -i elasticsearch /var/log/bitbucket/atlassian-bitbucket.log
2020-02-10 06:57:30,689 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2020-02-10 06:57:45,800 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 06:58:45,820 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:00:45,834 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:04:45,849 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:12:45,980 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:22:45,993 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:32:46,006 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:42:46,019 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:52:46,038 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:02:46,052 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:12:46,066 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
[root@bitbucket buckler]# systemctl status bitbucket
● bitbucket.service - Atlassian Bitbucket Server Service
Loaded: loaded (/usr/lib/systemd/system/bitbucket.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-10 06:56:15 UTC; 1h 25min ago
Process: 15562 ExecStop=/opt/bitbucket/bin/stop-bitbucket.sh (code=exited, status=0/SUCCESS)
Process: 15622 ExecStart=/opt/bitbucket/bin/start-bitbucket.sh (code=exited, status=0/SUCCESS)
Main PID: 15709 (java)
CGroup: /system.slice/bitbucket.service
└─15709 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m...

Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: - Access is allowed to 'others'. It is recommended that 'others' be denied
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: all access for security reasons.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: configured in _start-webapp.sh
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The current open files limit is set to less than 4096
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Attempting to increase limit...
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Limit increased to 4096 open files
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Starting Bitbucket webapp at http://localhost:7990
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The Bitbucket webapp has been started.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com systemd[1]: Started Atlassian Bitbucket Server Service.

 

The main problem here is how to start elasticsearch?

Best Regards, //Antonio

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

📣 Calling Bitbucket Data Center customers to participate in research

Hi everyone, Are you Bitbucket DC customer? If so, we'd love to talk to you! Our team wants to dive deep to understand your long-term plans regarding Bitbucket DC and Atlassian Cloud. Do you plan...

127 views 2 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you