I am trying to move my bitbucket pipelines to self hosted runners. I needs ssh keys. In hosted runners ssh key is mounted automatically. With self hosted runners, I did not find my id_rsa. So I added my private key and mounted it to my dind container and runner container (I can see the keys when I ssh into the pods). However when I run a pipeline that volume mount does not exist. How do I mount kubernetes secret into a self hosted pipeline runner?
If somebody comes here with the same problem, I recently found a bug, that sill exists in v1.326
We were getting an error, while accessing our private repo during the pipeline build, but only on local runners, cloud was working
The error was something like this:
Load key /tmp/<uuid>/ssh/id_rsa: invalid format
The answer ist, that the added ssh-key under settings, need to have a newline at the end, without one, the local runners aren't able to load the ssh-key as mentioned here.
Hopefully I helped someone, with this tip :)
@Edwin Knese we had exactly the same issue as you. I've tried the Kubernets and VM Linux runners and in both cases there was that problem.
On public runners the issue doesn't exist with the exactly same SSH key so BitBucket doing some adjustment on their machines.
Problem still exist on v1.363 runner.
@Caroline R please open the internal ticket to fix it or at least put some comments in runners configuration because there is so much waste time to looking solution for this (and from @Edwin Knese comments I assume is a known issue).
Thanks
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Edwin Knese and @Przemyslaw Nowak,
I am unable to reproduce the issue you are reporting. If you're still experiencing this, please create a new question via https://community.atlassian.com/t5/forums/postpage/board-id/bitbucket-questions and provide some more details like the following, and we can look into this
- what type of runners you are using (Linux Docker or Linux Shell) and what version
- whether you are adding SSH keys via Repository settings > SSH keys or if you're using Repository variables
- your bitbucket-pipelines.yml configuration
Kind regards,
Theodora
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Edwin Knese Wow, you really helped me. I added the newline to the key and suddenly it is working!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I have the same problem with runner version: 1.561.
We want to make a git submodule update and get this error:
Load key "/tmp/bitbucket-runner-72/028f872c-e360-5430-9ee8-af9bb1904f7b/ssh/id_rsa": invalid format
I looked at the private key that is located there and it seems like the line break at the end is missing.
So how did you add those new line at the end? In the bitbucket UI i'm not able to change the private key and in the pipeline i'm not able to access those file.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Okay i just had to delete the old ssh key in the bitbucket ui and create a new one. the new public key has a different format, seems like bitbucket internally a new type of key is generated.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi, @[deleted]!
Thank you for reaching out to Atlassian Community!
About the first part of your question, if you are referring to the ssh from Repository settings > SSH keys under the Pipelines section, then I would like to ask you to confirm if your Runner is up to date with the latest version, as we have released support for SSH keys and known hosts on runners in latest versions.
In order to upgrade it, you can stop the runner and use the below command to get the last version:
docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1
Then, about the second part of your question, when you say you ran a pipeline and that volume mount didn’t exist, and how to mount Kubernetes secret into a self-hosted pipeline runner, could you please give us more details about your request? You can share your configuration with us, so we can properly assist you.
Looking forward to hearing from you.
Kind regards,
Caroline
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks for the reply. So yes I am referring to the ssh key in the ssh keys pipelines.
This is the image I used:
docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
I imagine that is the latest.
if I run the pipeline via bitbucket I see the ssh key in
apiVersion: batch/v1
kind: Job
metadata:
name: runner
spec:
template:
metadata:
labels:
accountUuid: FOO
runnerUuid: FOO
repositoryUuid: FOO
spec:
containers:
- name: bitbucket-k8s-runner
image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
env:
- name: ACCOUNT_UUID
value: "FOO"
- name: RUNNER_UUID
value: "FOO"
- name: REPOSITORY_UUID
value: ""FOO""
- name: OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
name: runner-oauth-credentials
key: oauthClientId
- name: OAUTH_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: runner-oauth-credentials
key: oauthClientSecret
- name: WORKING_DIRECTORY
value: "/tmp"
volumeMounts:
- name: tmp
mountPath: /tmp
- name: docker-containers
mountPath: /var/lib/docker/containers
readOnly: true
- name: var-run
mountPath: /var/run
- name: ssh-key
readOnly: true
mountPath: /tmp/ssh
- name: docker-in-docker
image: docker:20.10.7-dind
securityContext:
privileged: true
volumeMounts:
- name: tmp
mountPath: /tmp
- name: docker-containers
mountPath: /var/lib/docker/containers
- name: var-run
mountPath: /var/run
- name: ssh-key
readOnly: true
mountPath: /tmp/ssh
restartPolicy: OnFailure
volumes:
- name: tmp
- name: docker-containers
- name: var-run
- name: ssh-key
secret:
secretName: ssh-key
backoffLimit: 6
completions: 1
parallelism: 1
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Olivier,
Please allow me to step in as Caroline is out of office.
The image you are using for the runner is the correct one, however, you may need to update it to the latest version. If you open the page of a Pipelines build on Bitbucket website and select a step that is running on your runner, select the word 'Runner' at the beginning of the log and you will see info about the Runner version. The latest one at the moment is 1.262.
If you are running an older version, you can upgrade it following the steps that Caroline mentioned in her previous reply. The latest version of the runner supports SSH keys that are added to a repo's Repository settings > SSH keys under the Pipelines section.
Please note that when you run a Pipelines build on your runner, the SSH key will not be located at /opt/atlassian/pipelines/agent/ssh/.
The path where the private SSH key is located includes the runner uuid which I'm afraid is not available during the build, and it will need to be hardcoded in your yml file. You can find it the following way:
- Run the command docker ps on the machine where the runner is running
- Find in the output the container for the Runner, and note its name, it should be something like runner-76b247e7-b925-5e7b-9da2-1cda14c4ff2c
For the example above, if the name of the runner is runner-76b247e7-b925-5e7b-9da2-1cda14c4ff2c and you have defined SSH keys in your Repository's settings and you are running the latest version of the runner, the private SSH key should be located at
/tmp/76b247e7-b925-5e7b-9da2-1cda14c4ff2c/ssh/id_rsa
If you have any questions, please feel free to let me know.
Kind regards,
Theodora
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.