git ssh to dockerized bitbucket server behind reverse nginx proxy (docker-compose)

[Daniel Einars]

Hi

I've been trying to set up the reverse proxy configuration with nginx and docker-compose but it keeps on asking me for a password (and even if I enter it correctly, refuses the connection) and I'm not sure where I'm going wrong at this point anymore.

I'm pretty sure I've missed something totally obvious but I think I've spent so much time on it I just can't see it anymore. If anyone could take a look I'd be incredibly happy to get that very last tid-bit working.

Here is my docker-compose and nginx configuration.

I've already tried a myriad of things, including enabeling ssh in the bitbucket serer and setting the url to default to one without a port number behind it. Any help would be greatly appreciated.

Update:

I've included some more information as suggested

Ah, thanks for pointing it out. I thought I had the domain names replaced.

the certbot certification is done once to get the certificates up and running and afterwards is accepted. I can access the bitbucket subdomain via web with encryption.

The problem I am having is trying to clone/push changes to a repository via git using ssh keys. I've enabled ssh and uploaded the public ssh key into the bitbucket server via the UI.

Inside the server configuration UI of bitbucket I've enabled ssh and set the base url to ssh://subdomain.domain.dev

I've created a sample project and added myself as the admin to that project. Then set created a local repository and set the remote url according to the instructions given in the bitbucket repository ui.

When I add a README.md, add all changes and

git push -u origin master -v

It reports that it is connecting to

git@subdomain.domaindev/project/repository.git

Then it asks me for a password. I haven't explicitly set a password (apart from my user password) but that one isn't being accepted either.

In the docker-compose file I realised last night that I didn't expose the nginx service on port 22. I'm not too familiar with how exactly the git ssh protocol works but is it something like

outgoing ssh:port -> my_vps_ip:22 -> docker-compose_nginx_service:22 -> bitbucket-server:22

Does bitbucket then forward the ssh port 22 to the standard port (7999 i think?)

Is there any other information or logs I could provide which might help?

[Daniel Einars · Accepted solution]

Right, so. After a long period of trying different settings I realized something very important.

SSH does not have the header information required for subdomain forwarding.

That means that you can't make a ssh request via the standard port to a subdomain. I will always go to the main domain and handled by the ssh agent there. I could open up a second port for ssh, but then I'd have the port number in the git@subdomain.domain.dev:PORT/project/repostiry which I specifically don't want.

I also don't want to have to specify a port number when sshing into my vps. So the only real solution would be to differentiate incoming requests based on their structure and forwarding them internally to the correct port.

Introducing SSLH. Basically:

 

sslh accepts connections on specified ports, and forwards them further based on tests performed on the first data packet sent by the remote client.

I haven't tried it yet but it sounds promising. There are some people using this when you google "nginx reverse proxy sslh". This, in my eyes, is the only viable solution at this point in time.

[Martin Melka]

Hi, did you get SSLH to work for your case? If you have a write-up, could you link it here? If not, can you at least confirm that this is a viable route to go?

[Daniel Einars]

Hi, 

I managed to get it going. The SSH works without a hitch these days.

Here are the relevant parts of the nginx.config and docker-compose-yml files

nginx.conf:

What did the trick was the stream enclosure in the end. It listens on Host port 22 and forwards to the docker container using the docker's inbuilt name-resolution to port 7999 (Bitbucket's default SSH port).

Also note the client-max-body-size in the bitbucket server config. This allows you to use git-lfs with larger files.

I've also included bits about ssl certs, ignore them if you want. The first server closure listens on port 80 and forwards anything to the appropriate docker container on port 443. 

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

stream {
    upstream ssh {
        server bitbucket-docker:7999;
    }
        server {
        listen        22;
        proxy_pass    ssh;
    }
}


http {
    include       /etc/nginx/mime.types;
    # default_type  application/octet-stream;
    default_type text/html;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;


    server {
        listen 80;
        server_name
            bitbucket.domain.dev www.bitbucket.domain.dev;
        server_tokens off;

        location /.well-known/acme-challenge/ {
            root /var/www/certbot;
        }
        location / {
            return 301 https://$host$request_uri;
        }
    }

    server {
        listen 443 http2 ssl;
        server_name bitbucket.domain.dev www.bitbucket.domain.dev;
        server_tokens off;

        ssl_certificate /etc/letsencrypt/live/domain.dev/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/domain.dev/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

        location / {
           client_max_body_size                256M;
            proxy_set_header Host               $host;
            proxy_set_header X-Real-IP          $remote_addr;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto  https;
            proxy_read_timeout                  60;
            proxy_connect_timeout               60;
            proxy_redirect                      off;
            proxy_pass                          http://bitbucket-docker:7990;
        }
    }


}

Here is my docker-compose.yml config. You can see that docker uses the container_name variable to do it's name resolution to find the right container in the nginx.config.

version: '3.7'

services:
  ### BEGIN PROXY/CERTBOT/SITE/CICD###
  nginx:
    image: nginx:1.15.9-alpine
    container_name: dle-nginx
    restart: unless-stopped
    volumes:
      - ./data/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    ports:
      - "80:80"
      - "443:443"
      - "22:22"
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
    depends_on:
      - certbot
      - bitbucket
  certbot:
    image: certbot/certbot:v0.33.1
    container_name: certbot-docker
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

  #  ### BEGIN BITBUCKET ###
  bitbucket:
    container_name: bitbucket-docker
    image: atlassian/bitbucket-server:6.3.0-ubuntu-jdk11
    restart: unless-stopped
    volumes:
      - bitbucket-data:/var/atlassian/application-data/bitbucket
    expose:
      - 7990
      - 7999
    environment:
      - SERVER_SECURE=true
      - SERVER_SCHEME=https
      - SERVER_PROXY_PORT=443
      - SERVER_PROXY_NAME=bitbucket.domain.dev
      - BITBUCKET_HOME=/var/atlassian/application-data/bitbucket/
    depends_on:
      - bitbucket-db
  bitbucket-db:
    image: postgres:9.6
    container_name: bitbucket-docker-db
    restart: unless-stopped
    expose:
      - 5000
    command: -p 5000
    volumes:
      - bitbucket-data-db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${BITBUCKET_DB_PASSWORD}
      - POSTGRES_USER=${BITBUCKET_DB_USER}
      - POSTGRES_DB=${BITBUCKET_DB}
      - PGDATA=/var/lib/postgresql/data/pgdata

volumes:
  bitbucket-data:
  bitbucket-data-db:

GOTCHAS:

While this works, I've run into permission errors on multiple occasions (i.e. Everything is set up and running, I've migrated projects and all seems fine for a couple of months. Then I updated some unrelated packages on the host and the Bitbucket-container crashes claiming it no longer has access rights to some files.)

I doubt this is specific to docker but it is mighty annoying.

Hope this helps!

[Martin Melka]

Thanks a lot for this writeup!

[Gonchik Tsymzhitov]

Hi! 

Also, could you  provide more details password for certbot or for connect ? 

Could you set into variables SERVER_PROXY_NAME , please? (it)

SERVER_PROXY_NAME=code.dle.dev

Cheers,

Gonchik Tsymzhitov

[Daniel Einars]

Ah, thanks for pointing it out. I thought I had the domain names replaced.

the certbot certification is done once to get the certificates up and running and afterwards is accepted. I can access the bitbucket subdomain via web with encryption.

The problem I am having is trying to clone/push changes to a repository via git using ssh keys. I've enabled ssh and uploaded the public ssh key into the bitbucket server via the UI.

Inside the server configuration UI of bitbucket I've enabled ssh and set the base url to ssh://subdomain.domain.dev

I've created a sample project and added myself as the admin to that project. Then set created a local repository and set the remote url according to the instructions given in the bitbucket repository ui.

When I add a README.md, add all changes and

git push -u origin master -v

 It reports that it is connecting to

git@subdomain.domaindev/project/repository.git

Then it asks me for a password. I haven't explicitly set a password (apart from my user password) but that one isn't being accepted either.

In the docker-compose file I realised last night that I didn't expose the nginx service on port 22. I'm not too familiar with how exactly the git ssh protocol works but is it something like

outgoing ssh:port -> my_vps_ip:22 -> docker-compose_nginx_service:22 -> bitbucket-server:22

Does bitbucket then forward the ssh port 22 to the standard port (7999 i think?)

Is there any other information or logs I could provide which might help?

[Daniel Einars]

Ah, thanks for pointing it out. I thought I had the domain names replaced.

the certbot certification is done once to get the certificates up and running and afterwards is accepted. I can access the bitbucket subdomain via web with encryption.

The problem I am having is trying to clone/push changes to a repository via git using ssh keys. I've enabled ssh and uploaded the public ssh key into the bitbucket server via the UI.

Inside the server configuration UI of bitbucket I've enabled ssh and set the base url to ssh://subdomain.domain.dev

I've created a sample project and added myself as the admin to that project. Then set created a local repository and set the remote url according to the instructions given in the bitbucket repository ui.

When I add a README.md, add all changes and

git push -u origin master -v

 It reports that it is connecting to

git@subdomain.domaindev/project/repository.git

Then it asks me for a password. I haven't explicitly set a password (apart from my user password) but that one isn't being accepted either.

In the docker-compose file I realised last night that I didn't expose the nginx service on port 22. I'm not too familiar with how exactly the git ssh protocol works but is it something like

outgoing ssh:port -> my_vps_ip:22 -> docker-compose_nginx_service:22 -> bitbucket-server:22

Does bitbucket then forward the ssh port 22 to the standard port (7999 i think?)

Is there any other information or logs I could provide which might help?

[Daniel Einars]

Ah, thanks for pointing it out. I thought I had the domain names replaced.

the certbot certification is done once to get the certificates up and running and afterwards is accepted. I can access the bitbucket subdomain via web with encryption.

The problem I am having is trying to clone/push changes to a repository via git using ssh keys. I've enabled ssh and uploaded the public ssh key into the bitbucket server via the UI.

Inside the server configuration UI of bitbucket I've enabled ssh and set the base url to ssh://subdomain.domain.dev

I've created a sample project and added myself as the admin to that project. Then set created a local repository and set the remote url according to the instructions given in the bitbucket repository ui.

When I add a README.md, add all changes and

git push -u origin master -v

It reports that it is connecting to

git@subdomain.domaindev/project/repository.git

Then it asks me for a password. I haven't explicitly set a password (apart from my user password) but that one isn't being accepted either.

In the docker-compose file I realised last night that I didn't expose the nginx service on port 22. I'm not too familiar with how exactly the git ssh protocol works but is it something like

outgoing ssh:port -> my_vps_ip:22 -> docker-compose_nginx_service:22 -> bitbucket-server:22

Does bitbucket then forward the ssh port 22 to the standard port (7999 i think?)

Is there any other information or logs I could provide which might help?