Hi,
We're noticing some issues when setting up bitbucket server, using the Amazon Linux based bitbucket AMI provided by atlassian (eu-west-2: ami-0a48a64baf846b803), on particular AWS EC2 instances.
We've reproduced this issue on a t3.medium, as well as an m5.large and an m5.xlarge (recommended in ATL docs)
The issue doesn't seem to occur on a t2.medium instance.
When we mount two EBS volumes, one for the root and another for the data, the data volume fails to properly initialise. Specifically, there's a failure during ZFS initialisation.
Relevant section from /var/log/atl.log
2019-12-06 11:45:15 Creating ZFS pool
missing link: nvme1n1 was partitioned but /dev/xvdf1 is missing
2019-12-06 11:45:46 === BEGIN: service atl-init-20-instance-store start ===
[...]
2019-12-09 11:45:06 Done disabling atl-init-30-db for future boots
2019-12-09 11:45:06 Ensuring /var/atlassian/application-data exists
2019-12-09 11:45:06 Atlassian application data mount /media/atl is not present. Creating /var/atlassian/application-data/db instead
Ultimately this failure means that bitbucket doesn't use the data volume we attached, and important data (repos in bitbucket home) are stored on the root volume.
This log line appears to correspond to the following code:
/etc/rc.d/init.d/atl-init-10-volume
atl_log "Creating ZFS pool"
# Create ZFS volume
zpool create tank $(basename ${ATL_APP_DATA_BLOCK_DEVICE}) >> "${ATL_LOG}" 2>&1
I'm still investigating this issue, but I suspect it's something to do with the "nitro" based instances, and how they handle mounting/linking nvme volumes.
I can repro the issue manually on the instance:
[root@ip-10-101-71-159 ec2-user]# export ATL_APP_DATA_BLOCK_DEVICE='/dev/xvdf'
[root@ip-10-101-71-159 ec2-user]# parted -s ${ATL_APP_DATA_BLOCK_DEVICE} mktable gpt
[root@ip-10-101-71-159 ec2-user]# zpool create tank ${ATL_APP_DATA_BLOCK_DEVICE}
missing link: nvme1n1 was partitioned but /dev/xvdf1 is missing
And if I fudge it by manually creating the expected symlink, it works:
[root@ip-10-101-71-159 ec2-user]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:1 0 50G 0 disk
├─nvme0n1p1 259:2 0 50G 0 part /
└─nvme0n1p128 259:3 0 1M 0 part
nvme1n1 259:0 0 100G 0 disk
├─nvme1n1p9 259:5 0 8M 0 part
└─nvme1n1p1 259:4 0 100G 0 part
[root@ip-10-101-71-159 dev]# ln -s /dev/nvme1n1p1 xvdf1
[root@ip-10-101-71-159 dev]# zpool create tank ${ATL_APP_DATA_BLOCK_DEVICE}
[root@ip-10-101-71-159 dev]# echo $?
0
[root@ip-10-101-71-159 dev]# zfs create -o compression=off -o sharesmb=off -o sharenfs=off -o atime=off -o recordsize=8k -o mountpoint=/media/atl tank/atlassian-home
[root@ip-10-101-71-159 dev]# mount -l
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=7965168k,nr_inodes=1991292,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/nvme0n1p1 on / type ext4 (rw,noatime,data=ordered) [/]
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tank on /tank type zfs (rw,xattr,noacl)
tank/atlassian-home on /media/atl type zfs (rw,noatime,xattr,noacl)
I could of course try creating the link as part of user data, but I'd rather not hack my way through this obstacle as this occurs using the recommended instance type, and there's got to be a proper way to fix this.
I've compared dmesg logs between the t2 (working) and t3 (not working) instances and there's an indication things are happening differently during the startup sequence (I see different software loaded when checking the disks, presumably because the t3 is nvme and the t2 isn't) but I have limited experience in this area so I haven't derived anything useful yet.
Has anybody run into this before, or otherwise knows what to do?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.