Pipeline unable to push to self-hosted docker registry

[vkost02]

I have a pipeline building a docker container, then pushing it to gitlab. This part works fine for many months now. I tried to add another step to push the same container to a self-hosted registry. This fails with 401 unauthorized, even though I know the credentials I supplied work. In fact, those credentials are the same as for the gitlab registry (I know this is not ideal but it should work regardless).

My self-hosted registry is protected by basic authentication, executed by a Traefik proxy, before the connection is passed on to the registry container. While analyzing Traefik logs, I noticed that there's a difference between my test pushes (made from my mac) and the push attempts made by the pipeline. My pushes have my username listed in the logs (CLF format), and the pipeline pushes appear as if they're not using the auth at all (no username in log entry, just "-"). 

I can provide more details if it would help. 

I appreciate any idea on how to tackle this - I am at a loss on how to proceed.

VKost

[vkost02]

Hm. It appears I found the problem, although I still do not fully understand why this is happening.

My self-hosted registry was set up to allow GET and HEAD request without credentials, and it required authentication for POST, PATCH, PUT, and DELETE. The idea was to allow docker pull to work without logging in, and request auth for docker push. Although this worked just fine in our tests (a few of us in the team successfully pulled and pushed from this registry) it did not work for the pipeline. Once I changed it, and required credentials for ALL OPERATIONS, things worked as expected. 

So if anyone knows why this is the case, I'd very much like to learn. Looking at my Traefik logs, it looks like docker attempts to connect without credentials, and if that fails, it uses the stored values. So, in my scenario, because GET was working with no credentials, docker seemed to think it doesn't need any for other operations. Weird, but not impossible. Perhaps there's a switch somewhere to turn this behavior off, and use the supplied credentials no matter what?

Thanks everyone