Azure Web App Git Deploy - Git Push not working

[Andreas Dasseville]

Hi,

I'm struggling to get GIT deploy working on an existing Azure web app. I did the complete same setup for another web app which is being used as a QA environment. The deploy to QA is an automated deploy for all commits on the master branch, and the commit to production is a manual deployment step and is the step after the QA deploy step.

I configured the deploy server (bitbucket pipelines using image microsoft/dotnet:sdk) and i'm doing these commands after the server pulled the sources to deploy:

• git init
• git config user.name "Bitbucket pipeline"
• git config user.email "bitbucket@doesnotexist.com"
• git add --all
• git remote add azure_web https://$AZ_DEPLOY_WEB_PRD_USR:$AZ_DEPLOY_WEB_PRD_PWD@$AZ_DEPLOY_WEB_PRD_GITURL
• git remote add azure_fct https://$AZ_DEPLOY_FCT_PRD_USR:$AZ_DEPLOY_FCT_PRD_PWD@$AZ_DEPLOY_FCT_PRD_GITURL
• git push azure_web master
• git push azure_fct master

The placeholders starting with $ signs are replaced by the deploy system for their real values set in a variables section.

I'm getting this exception: 

! [remote rejected] master -> master (shallow update not allowed)

I solved this by googling around and applying a fix found on a forum somewhere:

• going to the kudu debug console
• Navigating to d:\home\site\repository
• And firing the command: git config --local --add receive.shallowUpdate true

Unfortunately this only changed the error message, because it changed to:

! [remote rejected] master -> master (unable to migrate objects to permanent storage)

I'm puzzled because again, I did completely the same thing for another azure webapp, of course the variables values are different but the steps and commands are the same. No complaints from kudu or git comming from the other environment. Also I doubt the shallow update exception is really what's wrong because the QA environment does not have the receive.shallowUpdate setting set.

I activated some trace logging regarding GIT and I found out that the git push command only results in a http POST of 4 bytes for the production environment, and 11165 for the QA environment. It seems git push is malfunctioning for some reason?

I was able to reproduce the same behaviour on my local pc, by replicating the commands the pipeline does during it's build setup steps and appending my own steps.

Does anyone have an idea where I should look next to get this fixed?

thanks

[Edmondas Girkantas]

I found that Bitbucket by default uses shallowed version of your git repository (probably because of performance). So there are different options, you can change your Azure (Kudu) settings to accept shallowed version or you can tell bitbucket to use full repository clone, for example:

[Ciro ALABRESE]

the clone property didn't work for me, still the same problem, and cannot tell Kudu to accept shallow copy when using .deployment script

[Edmondas Girkantas]

Yeah, it would be pretty hard then to understand where's the problem.

Can you push that code from you local computer to the azure repository?

Please check if push url is correct, you can find it here YOUR_APP_NAME.scm.azurewebsites.net/api/scm/info

[Edmondas Girkantas]

@Ciro ALABRESE maybe in your deployment script or application settings you can increase diagnostic level for scm commands to find out what causes the issue that you have?

SCM_TRACE_LEVEL=4

[Ciro ALABRESE]

@Edmondas Girkantasthank you for your reply.

I already successfully push the code from my local computer. I will try this week to increase the diagnostic level and I'll let you know

[Ana Retamal]

Hi Andreas! Looks like the error you're getting is caused by Git, and not by the Pipelines. I've searched for the error "unable to migrate objects to permanent storage" and this is the explanation:

receive-pack: quarantine objects until pre-receive accepts

When a client pushes objects to us, index-pack checks the objects themselves and then installs them into place.
If we then reject the push due to a pre-receive hook, we cannot just delete the packfile; other processes may be depending on it. We have to do a normal reachability check at this point via git gc.

But such objects may hang around for weeks due to the gc.pruneExpire grace period. And worse, during that time they may be exploded from the pack into inefficient loose objects.

Instead, this patch teaches receive-pack to put the new objects into a "quarantine" temporary directory.
We make these objects available to the connectivity check and to the pre-receive hook, and then install them into place only if it is successful (and otherwise remove them as tempfiles).

Presumably when it compresses objects it puts them somewhere temporarily, then in a separate action tries to move them to their permanent location. Moving them means copying them from the temporary location to the permanent location, then either deleting them from the temporary location or leaving them to be removed by other processing, possibly at another time. 

The failure looks like a lack of permission (or disk space) to write to the remote location.

What git version are you using on the Server side? A git 2.10 would likely make that error disappear.

Hope that helps!

Ana

[Andreas Dasseville]

Hey,

Thanks for your help. I found that explanation too, but don't really understand how this would be applicable. The destination store has enough space left, and should not have permission issues. The destination is an Azure Webapp using the Kudu toolset which also provides a way to connect and deploy via GIT. For testing i created an environment from scratch to be sure every setting was set to the default value.

The GIT version is  2.14.1.windows.1

Thanks

[Nicholas Major]

@Andreas Dasseville Did you ever figure this out? I'm having the same issue.

[Steve Land]

I've had this issue for a few months - also only when running via pipelines (I can run the identical command fine from my machine). Annoying because I have to manually swap remotes and make the git push to Azure from my local dev machine after CI completes tests - not a great CI workflow.

Its worth adding this related link - https://stackoverflow.com/questions/42214667/what-does-git-mean-by-unable-to-migrate-objects-to-permanent-storage

My very rough guess is that its linux user permission related, but since I dont think we can sudo in pipelines it would be great if someone from the pipelines team could have a look into this! @Ana Retamal ?

If anyone finds a solution, please post it here :)

[Andreas Dasseville]

Sorry @Nicholas Major, I moved to another solution & platform. (Zip deploy to kudu using bamboo). Zip deploy will probably also work with pipelines but during the process of finding an alternative we also decided to use MSI authentication to connect to kudu which was already setup for our bamboo server.