Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Bitbucket Server setup doesn't detect data volume for some AWS EC2 instance types? Edited

Hi,

We're noticing some issues when setting up bitbucket server, using the Amazon Linux based bitbucket AMI provided by atlassian (eu-west-2: ami-0a48a64baf846b803), on particular AWS EC2 instances.

We've reproduced this issue on a t3.medium, as well as an m5.large and an m5.xlarge (recommended in ATL docs)

The issue doesn't seem to occur on a t2.medium instance.

When we mount two EBS volumes, one for the root and another for the data, the data volume fails to properly initialise. Specifically, there's a failure during ZFS initialisation.

Relevant section from /var/log/atl.log

2019-12-06 11:45:15 Creating ZFS pool
missing link: nvme1n1 was partitioned but /dev/xvdf1 is missing
2019-12-06 11:45:46 === BEGIN: service atl-init-20-instance-store start ===
[...]
2019-12-09 11:45:06 Done disabling atl-init-30-db for future boots
2019-12-09 11:45:06 Ensuring /var/atlassian/application-data exists
2019-12-09 11:45:06 Atlassian application data mount /media/atl is not present. Creating /var/atlassian/application-data/db instead

Ultimately this failure means that bitbucket doesn't use the data volume we attached, and important data (repos in bitbucket home) are stored on the root volume.

This log line appears to correspond to the following code:

/etc/rc.d/init.d/atl-init-10-volume

atl_log "Creating ZFS pool"

# Create ZFS volume
zpool create tank $(basename ${ATL_APP_DATA_BLOCK_DEVICE}) >> "${ATL_LOG}" 2>&1

I'm still investigating this issue, but I suspect it's something to do with the "nitro" based instances, and how they handle mounting/linking nvme volumes.

I can repro the issue manually on the instance:

[root@ip-10-101-71-159 ec2-user]# export ATL_APP_DATA_BLOCK_DEVICE='/dev/xvdf'
[root@ip-10-101-71-159 ec2-user]# parted -s ${ATL_APP_DATA_BLOCK_DEVICE} mktable gpt
[root@ip-10-101-71-159 ec2-user]# zpool create tank ${ATL_APP_DATA_BLOCK_DEVICE}
missing link: nvme1n1 was partitioned but /dev/xvdf1 is missing

And if I fudge it by manually creating the expected symlink, it works:

[root@ip-10-101-71-159 ec2-user]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:1 0 50G 0 disk
├─nvme0n1p1 259:2 0 50G 0 part /
└─nvme0n1p128 259:3 0 1M 0 part
nvme1n1 259:0 0 100G 0 disk
├─nvme1n1p9 259:5 0 8M 0 part
└─nvme1n1p1 259:4 0 100G 0 part
[root@ip-10-101-71-159 dev]# ln -s /dev/nvme1n1p1 xvdf1
[root@ip-10-101-71-159 dev]# zpool create tank ${ATL_APP_DATA_BLOCK_DEVICE}
[root@ip-10-101-71-159 dev]# echo $?
0
[root@ip-10-101-71-159 dev]# zfs create -o compression=off -o sharesmb=off -o sharenfs=off -o atime=off -o recordsize=8k -o mountpoint=/media/atl tank/atlassian-home
[root@ip-10-101-71-159 dev]# mount -l
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=7965168k,nr_inodes=1991292,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/nvme0n1p1 on / type ext4 (rw,noatime,data=ordered) [/]
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tank on /tank type zfs (rw,xattr,noacl)
tank/atlassian-home on /media/atl type zfs (rw,noatime,xattr,noacl)

I could of course try creating the link as part of user data, but I'd rather not hack my way through this obstacle as this occurs using the recommended instance type, and there's got to be a proper way to fix this.

I've compared dmesg logs between the t2 (working) and t3 (not working) instances and there's an indication things are happening differently during the startup sequence (I see different software loaded when checking the disks, presumably because the t3 is nvme and the t2 isn't) but I have limited experience in this area so I haven't derived anything useful yet.

Has anybody run into this before, or otherwise knows what to do?

1 answer

0 votes

Hi Laith 

Thank you for the question. The AMIs ami-0a48a64baf846b803 and ami-0773a7950c78a347a have not been yet tested with Nitro instances.

I have created a feature request for Bitbucket team to provide a support for nitro instances: https://jira.atlassian.com/browse/BSERV-12093

Kind Regards

Anton Shaleev

Atlassian Support EMEA  

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Powering DevOps with Bitbucket Server & Data Center

Hi everyone, The Cloud team recently announced 12 new DevOps features that help developers ship better code, faster   ! While we’re all excited about the new improvements to Bitbucket ...

1,892 views 0 7
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you