Bitbucket HA not working on Ubuntu-16

OS: Ubuntu-16 LTS
Bitbucket server: 4.11.0
Openstack VMs
Openstack Juno (Version: 2014.2.3) 

I am setting up Bitbucket in COLD STANDBY HA following this guide:
https://confluence.atlassian.com/bitbucketserver045/administering-bitbucket-server/advanced-actions/high-availability-for-bitbucket-server

I have external postgres DB

FIRST NODE

Brought up first node, got the error
"Required plugin com.atlassian.stash.ssh-plugin has failed to start"
Tried all solutions here:
https://confluence.atlassian.com/bitbucketserverkb/required-plugin-com-atlassian-stash-ssh-plugin-has-failed-to-start-is-displayed-when-accessing-the-bitbucket-server-web-interface-779171299.html

Did not help. Scrapped it and re-deployed. Now everything came up.

 

SECOND NODE

Installed Bitbucket server.
Made sure service is stopped.
Synced contents  of $HOMEDIR only (not $INSTALLDIR) from FIRST NODE to this node
Stopped atlbitbucket service on FIRST NODE 
Started atlbitbucket service on SECOND NODE 

I've now tried maybe 10 times and either I get this
"Required plugin com.atlassian.stash.ssh-plugin has failed to start"

or BLANK PAGE when trying to access Bitbucket web UI

Also in between scrap and re-install, even though Bitbucket service is stopped, it still holds onto the ports and shows it is listening on some of the various ports that it opens.

 

Any ideas on how to fix these problems?

Could this be related to Bitbucket not working well with SystemD, if so is it better to revert to Ubuntu-14 ?

 

Thanks

7 answers

0 votes

Hi John, 

in order to help you troubleshoot this, it would help if you attached some of the log messages from the atlassian-bitbucket.log when Bitbucket fails to start cleanly.

Re: systemd, Bitbucket Server will work with systemd, provided the limits are set correctly. You may run into issues scaling a production instance with Ubuntu's default limits under heavy load. One such limit Bitbucket Server may hit is the TasksMax limit, which in Ubuntu 16 LTS is, I believe, 512.

Felix

Felix, thank you. Can you please advise how to set this and/or any other limits for Bitbucket?

0 votes

You have 2 options for the TasksMax issue: either disable systemd's TasksAccounting, which keeps track of the number of tasks a service may spawn, or set TasksMax to a higher limit. 

Both of these changes are made in the systemd unit definition file for Bitbucket (/etc/systemd/system/atlbitbucket.service).

In the [Service] section, add either: 

TasksAccounting=false

or

TasksMax=12288

 

Felix

Kindly appreciated. Thanks Felix!

Felix, I was just looking into this and by default Bitbucket installer doesn't create atlbitbucket.service file in /etc/systemd/system or /lib/systemd/system

If I create the file with these permissions
-rwxr-xr-x 1 root root   32 Nov 14 16:31 atlbitbucket.service

cat atlbitbucket.service
[Service]
TasksAccounting=false

 
systemctl status atlbitbucket

● atlbitbucket.service
   Loaded: error (Reason: Invalid argument)
   Active: active (exited) since Mon 2016-11-14 16:31:07 UTC; 3min 14s ago

Nov 14 16:32:16 bitbucket-server.local systemd[1]: atlbitbucket.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

Nov 14 16:32:16 bitbucket-server.local systemd[1]: atlbitbucket.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

 

Is this a bug and should the installer be creating this file?

0 votes

Hi John, 

the installer does indeed not create the systemd service. There is an outstanding issue to provide systemd unit files in the Bitbucket Server distribution (https://jira.atlassian.com/browse/BSERV-9148).

From your original question I assumed you had created these yourself (you can check Google for a valid example of a Bitbucket Server systemd unit file). I believe systemd can and will generate unit files on the fly for init.d services, which is, I think, how you are currently launching Bitbucket. This means, however that you can't apply custom properties to the service (such as TasksAccounting). 

If you wish to use systemd with custom properties, you will need to manually configure the unit file until the aforementioned issue (BSERV-9148) is resolved, or apply the property to the systemd defaults.

Felix

Suggest an answer

Log in or Join to answer
Community showcase
Piotr Plewa
Published Dec 27, 2017 in Bitbucket

Recipe: Deploying AWS Lambda functions with Bitbucket Pipelines

Bitbucket Pipelines helps me manage and automate a number of serverless deployments to AWS Lambda and this is how I do it. I'm building Node.js Lambda functions using node-lambda&nbsp...

662 views 0 4
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot