Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,359,862
Community Members
 
Community Events
168
Community Groups

Bitbucket Server pull request page times out after moving home directory to nfs mount in CentOS 7

We recently moved the home drive for Bitbucket Server to an NFS mount for easier expandability and data recovery. Shortly after, users started reporting performance problems. The page times out when trying to look at diffs on a pull request page or just waiting for the "Merge" button to resolve.

At the times people complain, I see the following in the profiler log:

 [2.8ms] - Page com.atlassian.bitbucket.user.UserService.findGroupsByUser(String,PageRequest)

 [4.1ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

 [148745.4ms] - void com.atlassian.bitbucket.pull.PullRequestService.streamChanges(PullRequestChangesRequest,ChangeCallback)

  [3.8ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

  [60.2ms] - Map com.atlassian.stash.internal.comment.InternalCommentService.countCommentsByLocation(CommentSearchRequest)

   [60.2ms] - Map com.atlassian.stash.internal.comment.CommentDao.countsByLocation(CommentSearchCriteria)

  [148325.4ms] - git: with pull request lock(603, 276)

   [2.0ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)

   [2251.5ms] - git: resolve effective diff(276@1)

    [311.2ms] - git: resolve merge-base(276@1)

     [302.2ms] - nio: /usr/local/bin/git merge-base 2ad034b1d2dc3626922821f15177f07ba98f13cf e53dcda7c018040f91de3d288ad135f78efff1a3

    [1684.3ms] - git: update refs(276@1)

     [343.4ms] - nio: /usr/local/bin/git update-ref stash-refs/pull-requests/276/from 2ad034b1d2dc3626922821f15177f07ba98f13cf

  [1.1ms] - InternalGitPullRequestCachedMerge com.atlassian.stash.internal.scm.git.pull.GitPullRequestCachedMergeDao.findByPullRequest(PullRequest)

  [343.5ms] - nio: /usr/local/bin/git diff-tree -C -r --format=%H 4a382d6f0ebe01aff3d7b7b68270bd531614a991 2ad034b1d2dc3626922821f15177f07ba98f13cf --
In a test environment, I changed the nfs mount options to the recommended for data center (I cannot find "server" recommendations).
rw,nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768,_netdev
IT told me they do not see any issue on the NFS side. I don't know what else to try. We can't go back to local VM storage. It is not sustainable for backup and recovery.
Any ideas would be very helpful. Thanks.

1 answer

1 accepted

0 votes
Answer accepted

Hi Edward,

Did you move the entire home directory to NFS or just the shared directory? Moving the entire home directory to NFS is strongly discouraged since caches reside in this directory, and having those on NFS defeats the purpose of the caches (since they're supposed to locally cache files from NFS for faster access).

Cheers,

Christian

Premier Support Engineer

Atlassian

The whole home directory.

Changing the nfs mount options to the following may have fixed it.

rw,vers=3,rsize=32768,wsize=32768,hard,tcp,lock,proto=tcp,timeo=600,retrans=5

But I would like to investigate your suggestion. Is there a guide? Do I just mount the nfs folder to "shared"?

Hi Edward,

Yep, you can just mount NFS to <BitbucketHome>/shared.

You can find more at https://confluence.atlassian.com/bitbucketserverm/latest/install-bitbucket-data-center-1096089986.html under Step 1. Configure file share mounts.

Cheers,

Christian
Premier Support Engineer

Atlassian

The link does not work for me but I found what I need. Thank you.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
VERSION
7.9.1
TAGS
Community showcase
Published in Bitbucket

Git push size limits are coming to Bitbucket Cloud starting April 4th, 2022

Beginning on April 4th, we will be implementing push limits. This means that your push cannot be completed if it is over 3.5 GB. If you do attempt to complete a push that is over 3.5 GB, it will fail...

3,452 views 3 10
Read article

Atlassian Community Events