Error on pipeline bitbucket, load key SSH - docker on runner

[javier]

I have a pipeline in butbucket with a runner with docker install on a VPS. When I run the pipeline it fails in the line “git pull origin master”.

I set SSH keys in SSH keys menu on bitbucket I’ve added the file authorized_keys on VPS and it gives this error.

Host bitbucket.org

UseKeychain yes

IdentityFile ~/.ssh/jga

Error as follows:

Load key "/home/debian/apps/test/1401a1c6-b767-5d08-8df5-6edc1e68ccdb/ssh/id_rsa": invalid format

git@bitbucket.org: Permission denied (publickey).

It seems as if I have to configure something inside the docker (runner). I don’t know!

Any ideas?

[Patrik S · Accepted solution]

Hello @jg ,

Welcome to Atlassian Community!

From the error you have shared, it seems like the key you have configured might be wrong formatted or invalid. In order to better understand your current setup, I would like to understand the following :

• Are you doing a "git pull origin" from a repository different than the one pipelines is running ?
• Have you added your own SSH keys to Repository Settings > SSH Keys or you have used the Generate keys option ? If you have used your own keys, could you please remove try  removing those keys and setup using the Generate Keys option ?
  You will be provided with a public key that you then need to add as an Access key to the Bitbucket repository you are trying to clone.

You are also welcome to take a look in the following documentations that has instructions on using SSH keys in Bitbucket Pipelines, and also the blog post which covers cloning a different bitbucket repo in your pipeline build :

• https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/#Use-SSH-keys-in-Bitbucket-Pipelines
• https://bitbucket.org/blog/cloning-another-bitbucket-repository-in-bitbucket-pipelines

If the above does not work, I kindly ask you to share the bitbucket-pipelines.yml file of your build so I can have a deeper analysis on your current build setup.

Let me know if you have any questions.

Thank you @jg

Kind regards,

Patrik S

[javier]

Hi Patrick,

Thanks for your help, I’ve done all you told me.

I generated SSH Keys from Repository Settings/SSH Keys and added the the .pub generated in that path (pipeline) to my personal settings/SSH-Keys and the pipeline is running now.

However, the pipeline it doesn’t pull the repo to the VPS’ path. I mapped the volume in the runner and change the WORKING DIRECTORY as well, but I can’t see changes on my VPS’ path. It’s as if changes only are updated inside the runner (docker) but don't reflect in the VPS. This is the result of the pipeline, the runner and the list of files pulled to the docker/runner.

Runner (some values of the vairables has been change for privacy):

docker container run -it -v /home/debian/apps/test:/home/debian/apps/test -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro -e ACCOUNT_UUID={SENSITIVE} -e REPOSITORY_UUID={SENSITIVE} -e RUNNER_UUID={SENSITIVE} -e RUNTIME_PREREQUISITES_ENABLED=true -e OAUTH_CLIENT_ID=SENSITIVE -e OAUTH_CLIENT_SECRET=SENSITIVE -e WORKING_DIRECTORY=/home/debian/apps/test --name runner-MY_RUNNER docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

pipeline (this line has been change for privacy: git@bitbucket.org:jg/test.git):

image: rust

   pipelines:

      branches:

         master:

             - step:

                   name: 'Deployment to development'

                   deployment: Staging

                   runs-on:

                      - self.hosted

                      - linux

                      - 'dev'

                   script:

                       - cd /home/debian/apps/test

                       - pwd

                       - git init

                       - git remote add origin git@bitbucket.org:jg/test.git

                       - ls -la

                       - git fetch

                       - git checkout master

                       - git pull --rebase

                       - git config pull.rebase true

                       - git config pull.ff only

                       - git pull origin master

                       - pwd

                       - ls -la

                       - echo "The deployment to dev done";

ls -la (pipeline):

+ ls -la

total 124

drwxr-xr-x 5 root root 4096 Apr 28 15:07 .

drwxr-xr-x 3 root root 4096 Apr 28 15:07 ..

-rw-rw-rw- 1 root root 153 Apr 28 15:07.env

drwxrwxrwx 8 root root 4096 Apr 28 15:07 .git

-rw-rw-rw- 1 root root 8 Apr 28 15:07 .gitignore

drwxr-xr-x 6 root root 4096 Apr 28 15:07 1b56c9ca-2404-53d6-9b1f-003bd120cbef

-rw-rw-rw- 1 root root 44446 Apr 28 15:07 Cargo.lock

-rw-rw-rw- 1 root root 464 Apr 28 15:07 Cargo.toml

-rw-rw-rw- 1 root root 683 Apr 28 15:07 README.md

-rw-rw-rw- 1 root root 1308 Apr 28 15:07 bitbucket-pipelines.yml

drwxrwxrwx 2 root root 4096 Apr 28 15:07 src

However, inside the VPS' path (/home/debian/apps/test) there is nothing inside or pulled.

Thanks

[Patrik S]

Hello @jg ,

Thank you for providing additional context.

The behaviour you reported is actually expected : when your pipeline triggers a runner to run your build, the runner container will create some temporary folders in the mounted volume (by default /tmp), where it will temporarily save things like artifacts, your yml file and also clone the repository. After the build is done, the runner container will cleanup all the files it has downloaded to run the build, so you have a fresh environment to run the next build steps. So this is why you found the mounted folder to be empty after your build was completed.

Having said that,my understanding is that you are trying to use runners as a way to deploy files to your VPS server, by preserving all the files runner downloaded during the build and I'm afraid this is not currently possible.

My suggestion is this case, if you would like to deploy your files to your VPS server, is to use one of the following pipes :

• Rsync-deploy - https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/rsync-deploy
• SCP deploy - https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/scp-deploy
• SFTP Deploy - https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/sftp-deploy

The above pipes have the option to work authenticating with SSH key, which you have already configured, and will give you the functionally from copying files/folders from within the container to a remote server, in this case your VPS server.

We have also a set of other pipes that can be used to deploy to specific services, such as aws. You can take a look at deployment pipes in the link below : 

• https://bitbucket.org/product/features/pipelines/integrations?category=deployment&

Hope that helps to clarify your questions. Let me know in case you have any doubt.

Thank you, @jg .

Kind regards,

Patrik S

[javier]

Hi Patrick,

Thanks for your anwser.

Ok  I understand now. All changes will be on the container (runner).

I run the scp-deploy and works. Then, I don't need the runner, do I?.

What I don't get Is how I get changes from the runner, If I use artifacts to be used in the next step, the path will be also on dist/, inside the container.

If I run the runner on the VPS, whats is it for?, if I can't get changes on the VPS using the runner (container in my infrastructure  with a volume). Sorry, I read some articles about the runner, but I don't get it.

If the main aim is to deploy and update changes from local to the remote machine, how can I use the runner to deploy my changes without using scp/sftp or rsync?.

Thanks

[Patrik S]

Hello @jg ,

The main objective of using self-hosted runners is to allow you to run builds in Pipelines on your own infrastructure, so you are not charged for the build minutes. If you don't use runners, the build will run in pipelines infrastructure and the build minutes will be charged from your workspace quota.

So the linux runners are basically a docker container running in your own infrastructure where we run the scripts you have configured in your build, using the docker image you have chosen. The runner will have the same behaviour as pipelines, meaning you can still use artifacts and they will be passed to the next steps just like when running in pipelines. What will not happen is the runner to persist the downloaded files (repository, artifacts, etc) to the local volume, as I previously mentioned, and to achieve that you will need your build script to copy the files to the desired location, and for that you have the option to use the pipes, such as scp-deploy and rsync-deploy, that has most of the configuration already done for you. So choosing whether to use runners or not is choosing if you want the build to run on your infrastructure or not.

In summary, the way you do deployment is defined by the script you are executing in your build, independently if you are using runners or not. So if you configure your build to deploy using scp-deploy pipe, you can either run it on a self-hosted runner or in the pipelines environment, the only difference here will be that using runners you will not be charged for build minutes.

Let me know if that clarifies your questions.

Thank you @jg ,

Kind regards,

Patrik S.

[javier]

Hi Patrick,

Thanks for your anwser. Totally clear.

It's better to use runners., it's free.

Kind regards,