Bitbucket Server pull request page times out after moving home directory to nfs mount in CentOS 7

[Edward Ryan]

We recently moved the home drive for Bitbucket Server to an NFS mount for easier expandability and data recovery. Shortly after, users started reporting performance problems. The page times out when trying to look at diffs on a pull request page or just waiting for the "Merge" button to resolve.

At the times people complain, I see the following in the profiler log:

 [2.8ms] - Page com.atlassian.bitbucket.user.UserService.findGroupsByUser(String,PageRequest)

 [4.1ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

 [148745.4ms] - void com.atlassian.bitbucket.pull.PullRequestService.streamChanges(PullRequestChangesRequest,ChangeCallback)

  [3.8ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

  [60.2ms] - Map com.atlassian.stash.internal.comment.InternalCommentService.countCommentsByLocation(CommentSearchRequest)

   [60.2ms] - Map com.atlassian.stash.internal.comment.CommentDao.countsByLocation(CommentSearchCriteria)

  [148325.4ms] - git: with pull request lock(603, 276)

   [2.0ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

   [2251.5ms] - git: resolve effective diff(276@1)

    [311.2ms] - git: resolve merge-base(276@1)

     [302.2ms] - nio: /usr/local/bin/git merge-base 2ad034b1d2dc3626922821f15177f07ba98f13cf e53dcda7c018040f91de3d288ad135f78efff1a3

    [1684.3ms] - git: update refs(276@1)

     [343.4ms] - nio: /usr/local/bin/git update-ref stash-refs/pull-requests/276/from 2ad034b1d2dc3626922821f15177f07ba98f13cf

  [1.1ms] - InternalGitPullRequestCachedMerge com.atlassian.stash.internal.scm.git.pull.GitPullRequestCachedMergeDao.findByPullRequest(PullRequest)

  [343.5ms] - nio: /usr/local/bin/git diff-tree -C -r --format=%H 4a382d6f0ebe01aff3d7b7b68270bd531614a991 2ad034b1d2dc3626922821f15177f07ba98f13cf --

In a test environment, I changed the nfs mount options to the recommended for data center (I cannot find "server" recommendations).

rw,nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768,_netdev

IT told me they do not see any issue on the NFS side. I don't know what else to try. We can't go back to local VM storage. It is not sustainable for backup and recovery.

Any ideas would be very helpful. Thanks.

[Christian Glockner · Accepted solution]

Hi Edward,

Did you move the entire home directory to NFS or just the shared directory? Moving the entire home directory to NFS is strongly discouraged since caches reside in this directory, and having those on NFS defeats the purpose of the caches (since they're supposed to locally cache files from NFS for faster access).

Cheers,

Christian

Premier Support Engineer

Atlassian

[Edward Ryan]

The whole home directory.

Changing the nfs mount options to the following may have fixed it.

rw,vers=3,rsize=32768,wsize=32768,hard,tcp,lock,proto=tcp,timeo=600,retrans=5

But I would like to investigate your suggestion. Is there a guide? Do I just mount the nfs folder to "shared"?

[Christian Glockner]

Hi Edward,

Yep, you can just mount NFS to <BitbucketHome>/shared.

You can find more at https://confluence.atlassian.com/bitbucketserverm/latest/install-bitbucket-data-center-1096089986.html under Step 1. Configure file share mounts.

Cheers,

Christian
Premier Support Engineer

Atlassian

[Edward Ryan]

The link does not work for me but I found what I need. Thank you.