Pushing Large Files and getting Memory Errors - Help

[Denis_Dyack]

Hi There,

We are starting a game development studios and have about 1T of data backups that we would like to add to a repository. 

We currently have a Linux (Ubuntu) BitBucker Server running on a virtual machine - this seems to be working fine.

When I try to push commits of large sizes within source tree, we are running into memory errors - any advice on how to fix this?

For example, we will be working on high-end engines like Crytek, and they recommend checking the entire engine into source integrity which 3-6 Gigs (thousands of files) alone not including some of the large source files that we need to deal with for artist and other creatives.  

Thank you.

Best regards,
Denis

[Mikael Sandberg · Accepted solution]

I would take a look at using Git LFS, developed by GitHub, Atlassian and others to handle large files. We use is to store large video files and database binaries in our repositories.

[Pale Rider · Accepted solution]

Thank you for the quick response.  I have LFS enabled for bitbucket - perhaps it is not enabled properly with source tree. 

[Mikael Sandberg]

The memory errors you see sounds like the large files have not been added to Git LFS, but still exist in the repository. Also, if the files were in the repository before you setup Git LFS, did you clean up the history of the repository to move those files out of it?

[Denis_Dyack]

All new to this - sorry.  This is a brand new setup - just trying to set up initial repositories from a clean install/directory.  I did set LFS check mark in bitbucket - using Sourcetree as the interface (where something might not be set properly - all fresh installs (of everything! :).  

Basically, there are very large backup directories with engines, source art, video etc - in various directories.  We want to make this all accessible to the team.

[Mikael Sandberg]

Okay, so you probably need to add those directories to Git LFS, this command will do that:

git lfs track "<path to backup directory>/"

This will add the directory and all files under it to Git LFS. In the directories that have files added to Git LFS you will see a .gitattributes file that keeps track on files added to Git LFS. The tutorial I linked to above is really good at describing what you need to do to set everything up and how to use it.

[Denis_Dyack]

Great thank you - is there a way to do this in SourceTree? 

We tried the command line and it says git "does not know LFS" command.  This is a fresh sourcetree with bitbucket install.

[Mikael Sandberg]

Not if you want to add whole directories. In Sourcetree you only have the option to add file types after you initialize the repository the repository (Repository > Git LFS > Initialize Repository).

[Denis_Dyack]

Ok thank you - we are trying to initialize the LSF directory now through command lines.  The Initialize Repository option is currently greyed out.

[Denis_Dyack]

Hi Mikael,

Thank you for your help.  With your help, we have finally understood how to do this correctly.  The interface with SourceTree messed us up for a bit, but we found a process by jumping between command line commands and SourceTree visuals to correctly set the right files (by extensions) to LFS and set the regular text files to use GIT.  We have now successfully pushed 5-6 Gigs worth of files and testing a 70 directory to bitbucket, and it seems to be working.  Next, we have a 1 T directors :p.  

Thanks again!

Best regards,
Denis