Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Deleted user
Level
0 / 0 points
Next:
badges earned

Your Points Tracker
Challenges
Leaderboard
  • Global
  • Feed

Badge for your thoughts?

You're enrolled in our new beta rewards program. Join our group to get the inside scoop and share your feedback.

Join group
Recognition
Give the gift of kudos
You have 0 kudos available to give
Who do you want to recognize?
Why do you want to recognize them?
Kudos
Great job appreciating your peers!
Check back soon to give more kudos.

Past Kudos Given
No kudos given
You haven't given any kudos yet. Share the love above and you'll see it here.

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

elasticsearch not running with bitbucket v6.10

bitbucket version: 6.10.0.52

As seen from logs and troubleshooting, elasticsearch is not running:

[root@bitbucket ~]# ps -ef |grep [e]lastic 
[root@bitbucket ~]# tail /data/bitbucket/log/search/bitbucket_search.log
[2020-02-05T16:17:05,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:17:35,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:05,813][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:35,814][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:05,815][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:35,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:58,488][INFO ][o.e.n.Node ] [bitbucket_bundled]stopping ...
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]stopped
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]closing ...
[2020-02-05T16:20:01,741][INFO ][o.e.n.Node ] [bitbucket_bundled]closed
[root@bitbucket ~]# telnet localhost 7992
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

from /data/bitbucket/log/atlassian-bitbucket.log:

2020-02-07 06:14:20,905 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation. 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: e593ff34-b85c-4fb7-ab94-faca3409cfe4

 Also test from website is not working, see attached picture:

image-2020-02-07-07-13-23-813.png

I tried curl commands, tried to reset password but can't do any of it since elasticsearch is not started.

What else could I try?

Best Regards, Antonio

2 answers

1 accepted

0 votes
Answer accepted

SOLVED: There was two issues:

1. wrong (root) permissions on elasticsearch folder: /opt/bitbucket/elasticsearch/

2. Broken index, solved by folowing rebuild index page.

This problem seems to be due to low disk space. Try increasing the disk space of the mount point on which data folder is located.

Hi,

thank you for response and help, but it is not disk space issue, it was just temporary there.... :(

[root@bitbucket buckler]# df -h /data/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 15G 5.3G 74% /

As mentioned, connection doesn't work from web interface like I posted in screenshot above. It is not "refused", just failed to establish connection (which is normal because elastic search is not running).

I tried to reset password but it doesn't help (retreived values from $BITBUCKET_HOME/shared/search/buckler/buckler.yml file)

Here is what is shown in log, service status and evidence there is no two processes running as it should when you grep for bitbucket:

 

[root@bitbucket buckler]# ps -ef |grep [b]itb
tdsuser 15709 1 18 06:56 ? 00:16:15 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/data/bitbucket/tmp -Djava.library.path=/opt/bitbucket/lib/native;/data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start

[root@bitbucket buckler]# grep -i elasticsearch /var/log/bitbucket/atlassian-bitbucket.log
2020-02-10 06:57:30,689 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2020-02-10 06:57:45,800 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 06:58:45,820 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:00:45,834 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:04:45,849 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:12:45,980 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:22:45,993 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:32:46,006 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:42:46,019 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:52:46,038 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:02:46,052 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:12:46,066 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
[root@bitbucket buckler]# systemctl status bitbucket
● bitbucket.service - Atlassian Bitbucket Server Service
Loaded: loaded (/usr/lib/systemd/system/bitbucket.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-10 06:56:15 UTC; 1h 25min ago
Process: 15562 ExecStop=/opt/bitbucket/bin/stop-bitbucket.sh (code=exited, status=0/SUCCESS)
Process: 15622 ExecStart=/opt/bitbucket/bin/start-bitbucket.sh (code=exited, status=0/SUCCESS)
Main PID: 15709 (java)
CGroup: /system.slice/bitbucket.service
└─15709 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m...

Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: - Access is allowed to 'others'. It is recommended that 'others' be denied
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: all access for security reasons.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: configured in _start-webapp.sh
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The current open files limit is set to less than 4096
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Attempting to increase limit...
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Limit increased to 4096 open files
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Starting Bitbucket webapp at http://localhost:7990
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The Bitbucket webapp has been started.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com systemd[1]: Started Atlassian Bitbucket Server Service.

 

The main problem here is how to start elasticsearch?

Best Regards, //Antonio

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Calling any interview participants for Bitbucket Data Center

Hi everyone,  We are looking to learn more about development teams’ workflows and pain points, especially around DevOps, integrations, administration, scale, security, and the related challeng...

485 views 5 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you