LFS file download/checkout using the Bamboo SourceCodeCheckout leads to missing/incorrect lfs files Edited

We´ve migrated an existing Repository to make use of GIT-LFS.
The Repository is hosted at BitBucket Cloud while the bamboo server (latest version) is self-hosted behind a secured connection (vpn).

We´re using the Bamboo Source Code Checkout Task.
Bamboo uses a cached repository (/build-dir/_git-repositories-cache/...) to fasten the build process.
The initial clone to this cached repository path seems to work, all lfs files are downloaded successfully (as seen in the bamboo build log).
We tested this multiple times by flushing the cached repository contents and restart the process.
As far as we understand the bamboo source code checkout process the next step is to checkout the sources in the bamboo build dir
(in our case /build-dir/FSCWEB-FSCWEBSYMFONY-SS). The source code checkout (using a git remote pointing to file://CACHED_REPO)
works but after downloading the LFS files, which seems to work:
>> Git LFS: (31838 of 31838 files) 2.40 GB / 2.40 GB

the logs shows errors:
>> error: cannot feed the input to external filter true
>> error: external filter true failed

After the sourcode checkout task is finished, the pure git files are availale.
But the lfs files are missing (not existing in the filesystem) or have a incorrect filesize (0 or are very low and incorrect size).

This leads to multiple problems:
- git status or git lfs status shows many modified or deleted files (all from the lfs storage)
- the build process does not work as the files are required by following build stages.

Our current solution is to remove all files from the bamboo build dir and do a manually git-lfs clone from the bitbucket cloud repo url.
This downloads all files successfully (correct size, no git working dir changes after the clone). Then we start the bamboo build plan.
We´ve not tested to manipulate lfs files yet.

The git-lfs clone from the bitbucket repository works without problems at multiple developer machines, so the problem seems to be located at bamboo.

Our main question:
- How to fix the described problem?

Further questions:
- how does the bamboo git-lfs checkout task work in detail?
- Is a git-lfs clone used?
- Why does bamboo change the git remote url to file://CACHED_DIR during build instead of using the original Bitbucket Cloud url?

3 answers

0 votes

Hi Tobias,

It looks like a git lfs configuration error. Can you paste the contents of the .gitconfig file?

Cheers,

Branden

Hi branden, 

thank you for your answer!
There´s no .gitconfig in the bamboo build dir, here´s the output from the global config:

$ git config --global --list
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true

As discribed earlier the clone/checkout works without any problems at many developer machines so the problem seems to be located at the bamboo server.

We did a lot of research and I still think the problem is located at the way how bamboo handles the cached repository.

The git origin and lfs endpoint points to the cached repository with git-lfs client "file://"

$ git lfs env
git-lfs/2.2.1 (GitHub; linux amd64; go 1.8.3)
git version 1.9.1

Endpoint=file:///mnt/bamboo_storage/bamboo/xml-data/build-dir/_git-repositories-cache/ ....

But as described at the github git-lfs website git-lfs has no "file://"  support (github.com/git-lfs/git-lfs/wiki/Tutorial)

>
Note: only http:///https:// endpoints are supported by the git-lfs client. There is currently no direct ssh:// or file://. This means you can NOT have a local file lfs repo like you can for git.
>

Bamboo lfs support is enabled (as described here: https://www.atlassian.com/blog/bamboo/new-features-bamboo-5-15). Can you explain in detail how the bamboo git-lfs support looks like. As long as bamboo changes the origin and lfs endpoint to "file://" it won´t work.

Thank you for your help
ToBe


Hi Tobias,

Could I ask if there has been an answer to this question?  If so, could someone post it here?

I notice the question is over a month old.  This seems like a true bug in Bamboo's Git LFS support.  I am trying to do the same thing with a large asset-heavy db and though, like you, the inital clone is good (and long!) subsequent pulls just pull pointer files.

Thanks,

Bob

Hey Bob,

we´ve got no answer from the atlassian support. They changed their policy and with a starter package you get no personal support any longer. Only this forum an as you see no help ;-(

I disabled the bamboo sourcecode checkout task and implemented a script based git checkout.


Suggest an answer

Log in or Sign up to answer
Community showcase
Posted Monday in Confluence

Organizing your space just got easier - Page Tree Drag & Drop is here

Hi Community! I’m Elaine, Confluence Product Manager. You may have read my earlier post about page tree in space navigation sidebar. I'm excited to share another improvement that helps you organize ...

72 views 3 3
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you