Using http to clone a git repo results in no files (but ssh works)

Question

We have a repo that hosts a Django / react framework. When checking it out over http, git returns complete success, but fails to extract any files, and removes everything.

$ git clone http://server:7990/scm/ad/repo.git
Cloning into 'repo'...
Username for 'http://server:7990': user
Password for 'http://user@server:7990':
remote: Enumerating objects: 787, done.
remote: Counting objects: 100% (787/787), done.
remote: Compressing objects: 100% (633/633), done.
remote: Total 787 (delta 449), reused 239 (delta 107)
Receiving objects: 100% (787/787), 3.09 MiB | 5.13 MiB/s, done.
Resolving deltas: 100% (449/449), done.
$ ls repo
/bin/ls: repo: No such file or directory

Doing the same thing over ssh works perfectly. Same behaviour in sourcetree/git command line windows/linux. Network and local.

tracing git with

export GIT_TRACE_PACKET=1
 export GIT_TRACE=1
 export GIT_CURL_VERBOSE=1

Adds no further information.

Logfile from bitbucket:

2018-11-13 12:13:23,377 DEBUG [http-nio-7990-exec-4] @1G0EXKKx733x59x0 127.0.0.1 "GET /scm/ad/repo.git/info/refs HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2018-11-13 12:13:23,378 DEBUG [http-nio-7990-exec-4] @1G0EXKKx733x59x0 127.0.0.1 "GET /scm/ad/repo.git/info/refs HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]
2018-11-13 12:13:27,092 DEBUG [http-nio-7990-exec-7] @1G0EXKKx733x60x0 127.0.0.1 "GET /scm/ad/repo.git/info/refs HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2018-11-13 12:13:27,151 DEBUG [http-nio-7990-exec-7] user @1G0EXKKx733x60x0 127.0.0.1 "GET /scm/ad/repo.git/info/refs HTTP/1.1" c.a.bitbucket.scm.BaseCommand Executed /usr/bin/git http-backend
2018-11-13 12:13:27,238 DEBUG [http-nio-7990-exec-1] @1G0EXKKx733x61x0 127.0.0.1 "POST /scm/ad/repo.git/git-upload-pack HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2018-11-13 12:13:28,245 DEBUG [http-nio-7990-exec-1] user @1G0EXKKx733x61x0 127.0.0.1 "POST /scm/ad/repo.git/git-upload-pack HTTP/1.1" c.a.bitbucket.scm.BaseCommand Executed /usr/bin/git http-backend
2018-11-13 12:13:28,596 INFO [io-pump:thread-8] user @1G0EXKKx733x61x0 127.0.0.1 "POST /scm/ad/repo.git/git-upload-pack HTTP/1.1" c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
2018-11-13 12:13:29,273 DEBUG [http-nio-7990-exec-1] user @1G0EXKKx733x61x0 127.0.0.1 "POST /scm/ad/repo.git/git-upload-pack HTTP/1.1" c.a.bitbucket.scm.BaseCommand Executed /usr/bin/git http-backend
2018-11-13 12:13:29,274 DEBUG [http-nio-7990-exec-1] user @1G0EXKKx733x61x0 127.0.0.1 "POST /scm/ad/repo.git/git-upload-pack HTTP/1.1" c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection

It must be something in the repository causing the issue with http cloning, but it seems odd that no error is generated.

Strace indicates the following:

open("/home/user/git/repo/.git/config", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0664, st_size=195, ...}) = 0
fstat(6, {st_mode=S_IFREG|0664, st_size=195, ...}) = 0
mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0c76261000
read(6, "[core]
	repositoryformatversion "..., 32768) = 195
read(6, "", 32768) = 0
close(6) = 0
munmap(0x7f0c76261000, 32768) 
write(4, "option check-connectivity true
", 31) = 31
read(3, "ok
", 4096) = 3
write(4, "option cloning true
", 20) = 20
read(3, "ok
", 4096) = 3
write(4, "fetch 9f519e41f8bb7af42f5863ddb6"..., 131) = 131
read(3, "", 4096) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=12384, si_status=SIGPIPE, si_utime=0, si_stime=1} ---
openat(AT_FDCWD, "repo/.git", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
getdents(6, /* 10 entries */, 32768) = 280
lstat("repo/.git/HEAD", {st_mode=S_IFREG|0664, st_size=23, ...}) = 0
unlink("repo/.git/HEAD") = 0
lstat("repo/.git/branches", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
openat(AT_FDCWD, "repo/.git/branches", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, /* 2 entries */, 32768) = 48
getdents(7, /* 0 entries */, 32768) = 0
close(7) = 0
rmdir("repo/.git/branches") = 0

9f519e41f8bb7af42f5863ddb6 being the latest check in at the time.

Christian Glockner · Answer

Hi Matthew,

if you're not using the latest version of git I'd upgrade it and see if that brings any improvement.

Cheers,

Christian

Premier Support Engineer

Atlassian

Matthew Parslow · Answer

Hi Christian,

That's right, it happens on localhost and remote connections, The only thing I can think of is that somehow the content is triggering it, given it works over ssh, and that other repositories work over http. (in fact the logs here are from a local connection)

This initially presented on a windows client, and we assumed there was a specific problem with windows and possible AV of some sort.

What appears to be happening is that it's requesting the latest change, The change returns no data for some reason.

write(4, "fetch 9f519e41f8bb7af42f5863ddb6"..., 131) = 131
read(3, "", 4096) = 0

The server returns no data, which causes the client to crash, which disconnects. Unfortunately this type of failure case isn't captured correctly in the client in terms of error messages, but it cleans itself up after the crash as if it never happened.

In fact I just retested, and it does the same thing (with a different change)

The existing clones of the repository using http were still working for push and pull.

I did a sync with the test repository and the return in this case is just a newline.

write(4, "fetch 46622219c49532b38926a985fd"..., 131) = 131
read(3, "
", 4096) = 1

This would indicate that's the expected behaviour - I haven't got a lot of experience with the git protocol, but this seems reasonable.

Unfortunately it's an internal repository I can't share publicly.

Christian Glockner · Answer

Hi Matthew,

What a curious case!

From the strace it looks like you're getting an SIGPIPE, i.e. a broken pipe, and in the Bitbucket logs this is correspondingly represented as "the remote client aborted the connection".

You said this happens when doing it locally too - do you mean you get the same results when you clone using "localhost" on the machine where Bitbucket Server is installed?

Cheers,

Christian

Premier Support Engineer

Atlassian