It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Binary reject control not working correctly

Devtools Barclays Aug 30, 2016

We have implemented the binary reject control and it works fine if we just push the code for the first time. BUt if we amend the commit to remove the file and push to BBS, it fails to stop the push . Also we saw the repo size before the push and after it , there was a size increase in the repo.Please let us know if you require any other info from our side.Thanks

Steps To Reproduce1. Note the size of the Git repo in the BBS user interface (Settings -> retrieve size)

 2. Commit a binary file to a Git repo 

3.Push to BBS, you will get a rejection message 

4.Amend your commit to remove the file from the commit (git rm binFile.exe; git commit --amend) 

5.Push the commit to BBS - it will pass Note the size of the Git repo in the user interface (Settings -> retrieve size) - it will be much larger 

6.Mirror clone the repository locally to a new location (git clone --mirror ...)

 7.Perform some analysis on the repo git rev-list --objects --all | sort -k 2 > allfileshas.txt git gc && git verify-pack -v objects/pack/pack-*.idx | egrep "^\w+ blob\W+[0-9]+ [0-9]+ [0-9]+\w" | awk '{print $1,$3}' | sort -k 2 -n -r > bigobjectsall.txt for SHA in `cut -f 1 -d\ < bigobjectsall.txt`; do echo $(grep ^$SHA bigobjectsall.txt) $(grep $SHA allfileshas.txt) >> bigtosmallall.txt done; Check the file bigtosmallall.txt You should see a line referring to the file without a path, eg. $ less bigtosmallall.txt 2491b7745f89d0cf89bcb601a850563e90e4a0a8 14296728 e96534fb27b68192f27f985d3879e173ec77adb8 8792 e96534fb27b68192f27f985d3879e173ec77adb .idea/uiDesigner.xml 

8.Proof that the binary file has actually been pushed to BBS and bypassed the control (although the file is not visible in the repo, it is still carried around in the commit history)

Script used:

// Exempt repository IDs
if (repository.id in [10337]) {
return false;
}

// Reject the following filetypes, except if the file is gradle-wrapper.jar
return pathsMatchExcludingDeletes('regex : (?i:.*
.(dll|exe|zip|tar|gz|pdb|msi)$)')

 

 

com.onresolve.jira.stash.groovyrunner

1 answer

1 vote
Adam Markham [Adaptavist] Community Leader Aug 30, 2016

From what I can see this is an issue with all pre-receive hooks. Bitbucket will collect the objects that are unreferenced when it does a full garbage collection, although the file does not appear in the commit history on Bitbucket.

This is an issue with all pre-receive hooks and is not limited to ScriptRunner.

Please see: https://confluence.atlassian.com/bitbucket/maintaining-a-git-repository-321848291.html

You can read more about git's garbage collector, and ways to manually invoke it at https://git-scm.com/docs/git-gc. Take note of the gc.reflogExpire and gc.reflogExpireUnreachable options in the Configuration section if you want to expire old objects more quickly.

You may want to take this up with Atlassian Support but don't mention ScriptRunner as they will just send you back to us. You'll probably see the same behaviour when you have no hooks and then commit and push a large file and then later remove it, the size of the repository will not reduce to reflect that until garbage collection kicks in.

I'm not saying this is a bug this is how Bitbucket and Git work, just Atlassian can provide more support on how to do the garbage collection etc.

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Published in Bitbucket Pipelines

Building a Bitbucket Pipe as a casual coder

...ipe.sh :  #!/bin/bash source "$(dirname "$0")/common.sh" enable_debug extra_args="" if [[ "${DEBUG}" == "true" ]]; then extra_args="--verbose" fi # mandatory variables R...

3,033 views 1 20
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you