nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Bitbucket Server pull request page times out after moving home directory to nfs mount in CentOS 7

Edward Ryan

We recently moved the home drive for Bitbucket Server to an NFS mount for easier expandability and data recovery. Shortly after, users started reporting performance problems. The page times out when trying to look at diffs on a pull request page or just waiting for the "Merge" button to resolve.

At the times people complain, I see the following in the profiler log:

 [2.8ms] - Page com.atlassian.bitbucket.user.UserService.findGroupsByUser(String,PageRequest)

 [4.1ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

 [148745.4ms] - void com.atlassian.bitbucket.pull.PullRequestService.streamChanges(PullRequestChangesRequest,ChangeCallback)

  [3.8ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

  [60.2ms] - Map com.atlassian.stash.internal.comment.InternalCommentService.countCommentsByLocation(CommentSearchRequest)

   [60.2ms] - Map com.atlassian.stash.internal.comment.CommentDao.countsByLocation(CommentSearchCriteria)

  [148325.4ms] - git: with pull request lock(603, 276)

   [2.0ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

   [2251.5ms] - git: resolve effective diff(2761)

    [311.2ms] - git: resolve merge-base(2761)

     [302.2ms] - nio: /usr/local/bin/git merge-base 2ad034b1d2dc3626922821f15177f07ba98f13cf e53dcda7c018040f91de3d288ad135f78efff1a3

    [1684.3ms] - git: update refs(2761)

     [343.4ms] - nio: /usr/local/bin/git update-ref stash-refs/pull-requests/276/from 2ad034b1d2dc3626922821f15177f07ba98f13cf

  [1.1ms] - InternalGitPullRequestCachedMerge com.atlassian.stash.internal.scm.git.pull.GitPullRequestCachedMergeDao.findByPullRequest(PullRequest)

  [343.5ms] - nio: /usr/local/bin/git diff-tree -C -r --format=%H 4a382d6f0ebe01aff3d7b7b68270bd531614a991 2ad034b1d2dc3626922821f15177f07ba98f13cf --

In a test environment, I changed the nfs mount options to the recommended for data center (I cannot find "server" recommendations).

rw,nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768,_netdev

IT told me they do not see any issue on the NFS side. I don't know what else to try. We can't go back to local VM storage. It is not sustainable for backup and recovery.

Any ideas would be very helpful. Thanks.

Accepted answers

Christian Glockner

Hi Edward,

Did you move the entire home directory to NFS or just the shared directory? Moving the entire home directory to NFS is strongly discouraged since caches reside in this directory, and having those on NFS defeats the purpose of the caches (since they're supposed to locally cache files from NFS for faster access).