LFS files marked as changed after clone in pipeline

[Matt Schuckmann]

I've got repo with 4 binary files store in LFS and when my pipeline clones this repo and runs a `git status` or `git diff` command sometimes 1 or more of the binary files are reported as modified, however when I check the files (via sha256sum) I can find no discernible difference.  Which files are reported as modified is totally random, it can be all 4 files or just 1 or 2 or none of them. 

So far this has only been observed with the pipeline build. The only other thing of note is that the pipeline is running as self hosted on my own hardware. 

I've checked that the files are properly marked as binary and lfs in the .gitattributes file. 

I'm really stumped on what is causing this, any help would be greatly appreciated. 

[Theodora Boudale]

Hi @Matt Schuckmann ,

I've seen this issue occur when the Docker image used as a build container (the one you define in your bitbucket-pipelines.yml file) doesn't have git lfs installed.

If this is the case, you can try:

• either using a different Docker image that has git lfs pre-installed
• or install git lfs at the beginning of each step

If you use a Docker image based on a Linux distribution that uses apt commands, you can install git lfs during the build by adding the following commands in your script:

- apt-get update
- curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
- apt-get install git-lfs

Then run a git status.

Please feel free to let me know how it goes.

Kind regards,
Theodora

[Matt Schuckmann]

Hi Theodora, 

Thanks for the response but the docker image for the pipeline already does have git-lfs installed. In fact we this same docker image for our development environment and we never see this problem when doing development, the problem only occurs in the pipeline. 

[Matt Schuckmann]

So I tried adding a `git stash && git drop` to my pipeline to clean up the dirty files status and the results are not good, it looks like after the stash the binary files are not correct at all (log below). 

Prior to running the `git stash && git drop` command `git status` and `git lfs status` shows 3 modified files, however the sizes sha256 checksums for the files look correct. 

After the running `git stash && git drop` the file sizes and checksums are all wrong but `git status` and `git lfs status `say there are no modified files. 

Runner labels: self.hosted, linux
Runner version:
current: 1.287
latest: 1.287
+ umask 000

+ retry 6 git clone --branch="feature/resolve-pipeline-dirty-issue" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...

+ git reset --hard 51ee15fd053bca2b24162e877f027d0720c7cced
HEAD is now at 51ee15f Debugging issue with dirty files in pipeline checkout

+ git config user.name bitbucket-pipelines

+ git config user.email commits-noreply@bitbucket.org

+ git config push.default current

+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/

+ git remote set-url origin http://bitbucket.org/$BITBUCKET_REPO_FULL_NAME

+ git reflog expire --expire=all --all

+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude

+ chmod 777 $BUILD_DIR

Images used:
build: public.ecr.aws/t5r0t2r4/vitis@sha256:36238d5ab11dba93ad0d56b2080f5a8a5d992f9033a69785522554a10b73fd60

+ git status
On branch feature/resolve-pipeline-dirty-issue
Your branch is up to date with 'origin/feature/resolve-pipeline-dirty-issue'.

Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)

modified: binaries/u-boot.elf
modified: design_1_xsa/design_1_wrapper.bit
modified: design_1_xsa/design_1_wrapper.xsa

no changes added to commit (use "git add" and/or "git commit -a")

+ git lfs status
On branch feature/resolve-pipeline-dirty-issue
Git LFS objects to be pushed to origin/feature/resolve-pipeline-dirty-issue:


Git LFS objects to be committed:


Git LFS objects not staged for commit:

binaries/u-boot.elf (LFS: 173ff22 -> File: 173ff22)
design_1_xsa/design_1_wrapper.bit (LFS: 67c784b -> File: 67c784b)
design_1_xsa/design_1_wrapper.xsa (LFS: 75a6c0d -> File: 75a6c0d)


+ ls -al binaries/
total 3348
drwxrwxrwx 2 root root 4096 Feb 2 17:56 .
drwxrwxrwx 7 root root 4096 Feb 2 17:56 ..
-rw-rw-rw- 1 root root 2791916 Feb 2 17:56 u-boot.bin
-rw-rw-rw- 1 root root 623804 Feb 2 17:56 u-boot.elf

+ ls -al design_1_xsa/
total 2540
drwxrwxrwx 2 root root 4096 Feb 2 17:56 .
drwxrwxrwx 7 root root 4096 Feb 2 17:56 ..
-rw-rw-rw- 1 root root 2083852 Feb 2 17:56 design_1_wrapper.bit
-rw-rw-rw- 1 root root 499397 Feb 2 17:56 design_1_wrapper.xsa
-rw-rw-rw- 1 root root 141 Feb 2 17:56 .gitignore
-rw-rw-rw- 1 root root 158 Feb 2 17:56 README.txt

+ sha256sum binaries/u-boot.bin
cdfa142f7e5e52290be684069a697a93ffedbd0c584519d308b4b9452e572573 binaries/u-boot.bin

+ sha256sum binaries/u-boot.elf
173ff2205e330dff69629584be6ce1d87b62645b5d7b88f95839c3f03c4b86b1 binaries/u-boot.elf

+ sha256sum design_1_xsa/design_1_wrapper.bit
67c784b4a9475169f53bcb259ded80d5baf1c44847920559e3c44ad0247e17f3 design_1_xsa/design_1_wrapper.bit

+ sha256sum design_1_xsa/design_1_wrapper.xsa
75a6c0d6e11d02300ad8c1e95fa6fdde64a0664c5b5207964ac051d270d34fb8 design_1_xsa/design_1_wrapper.xsa

+ git stash
Saved working directory and index state WIP on feature/resolve-pipeline-dirty-issue: 51ee15f Debugging issue with dirty files in pipeline checkout

+ git stash drop
Dropped refs/stash@{0} (96d42b1a27f8979ea4d1c48f507aceced5e66179)

+ git status
On branch feature/resolve-pipeline-dirty-issue
Your branch is up to date with 'origin/feature/resolve-pipeline-dirty-issue'.

nothing to commit, working tree clean

+ git lfs status
On branch feature/resolve-pipeline-dirty-issue
Git LFS objects to be pushed to origin/feature/resolve-pipeline-dirty-issue:


Git LFS objects to be committed:


Git LFS objects not staged for commit:

 

+ ls -al binaries/
total 2740
drwxrwxrwx 2 root root 4096 Feb 2 17:56 .
drwxrwxrwx 7 root root 4096 Feb 2 17:56 ..
-rw-rw-rw- 1 root root 2791916 Feb 2 17:56 u-boot.bin
-rw-rw-rw- 1 root root 131 Feb 2 17:56 u-boot.elf

+ ls -al design_1_xsa/
total 24
drwxrwxrwx 2 root root 4096 Feb 2 17:56 .
drwxrwxrwx 7 root root 4096 Feb 2 17:56 ..
-rw-rw-rw- 1 root root 132 Feb 2 17:56 design_1_wrapper.bit
-rw-rw-rw- 1 root root 131 Feb 2 17:56 design_1_wrapper.xsa
-rw-rw-rw- 1 root root 141 Feb 2 17:56 .gitignore
-rw-rw-rw- 1 root root 158 Feb 2 17:56 README.txt

+ sha256sum binaries/u-boot.bin
cdfa142f7e5e52290be684069a697a93ffedbd0c584519d308b4b9452e572573 binaries/u-boot.bin

+ sha256sum binaries/u-boot.elf
95c596298a15df568cb6d775d53d6ade14424e9aee5ad954253e8b8f54838dff binaries/u-boot.elf

+ sha256sum design_1_xsa/design_1_wrapper.bit
4be8146a185b8597bfccb501e7f3a78ba0b88c777c40e53f757fa8203182ce57 design_1_xsa/design_1_wrapper.bit

+ sha256sum design_1_xsa/design_1_wrapper.xsa

2de1ca29632e4caf9bc6eed478583a9ba9f99fb1e68ff0ef4c90f9d63886beb1 design_1_xsa/design_1_wrapper.xsa

[Theodora Boudale]

Hi Matt,

Thank you for your reply.

1. If you run the following commands on a machine of yours that has Git LFS installed

git clone --branch="feature/resolve-pipeline-dirty-issue" --depth 50 <REPO_URL>
git reset --hard 51ee15fd053bca2b24162e877f027d0720c7cced
git status

do you see files as modified in the output of git status or not? I'm asking this so we can narrow down whether this may be a repo-specific issue or not.

2. Have you always been tracking the files listed in .gitattributes with Git LFS in this repo, or were such files previously tracked by Git and you migrated them to Git LFS later (e.g. with BFG or git lfs migrate)?

Kind regards,
Theodora

[Matt Schuckmann]

Hi Theodora, 

I think we've resolved the problem by installing the latest version of git and git-lfs in the docker container running the pipeline. The docker container is based on Ubuntu 18.04 with git and git lfs installed from apt packages so the versions were pretty old.  I went ahead and installed both from source and so far the problem has not come back. 

It's still not clear to me why we only saw the problem when running from the pipeline. I suppose a version mismatch between the git that does the clone and the git that checks the status could cause the problem, is that possible.

Here is a thread from the git-lfs github that helped me 
https://github.com/git-lfs/git-lfs/issues/3787

Matt S. 

[Theodora Boudale]

Hi Matt,

Thank you for the update, It's good to hear that you have resolved the issue.

I'm not sure if the version mismatch you mention was the issue. Considering that git lfs --version was not showing the version of Git LFS, I would assume that it might have been a bug with the Git LFS version that was previously installed in the Docker container.

Kind regards,
Theodora