Deploy repository using SSH and Pipelines

[EVO_9]

The documentation on Pipelines is shocking and nearly every post I have read, the user has had issues setting this up and ended up moving to Jenkins which gets recommended a lot. I would like to keep this in-house seeing as I'm already using Bitbucket.

I have successfully added my key to Bitbucket which allows me to post my local files to my Bitbucket repository.

I then enable Pipelines, create an environment variable called MY_SSH_KEY with my public ssh key, I also encoded it using the following link and pasted it into the variable.

I just can't get this to work and I'm on the verge of jumping ship as well as I've wasted a week on this.

Here's my Pipeline code

pipelines: 
    default: 
        - step: 
            script:  
                - mkdir -p ~/.ssh
                    - cat my_known_hosts >> ~/.ssh/known_hosts
                    - (umask  077 ; echo $MY_SSH_KEY | base64 --decode -i > ~/.ssh/id_rsa)
                    - ssh user@domain 'bash -s' < ./deploy.bash

The error I'm getting is:

+ cat my_known_hosts >> ~/.ssh/known_hosts

bash: /opt/atlassian/pipelines/agent/tmp/bashScript4585730180615321479.sh: line 16: syntax error near unexpected token `;&'

Can someone please help me to do this and not just paste a link  as it's more than likely I have already read the link.

[Philip Hodder]

Hi Daniel,

It looks like you've found some old Community questions. Sorry, we've forgotten to keep these responses up to date.

If you tell me what you're specifically trying to do I can give you a more catered answer. I've posted an updated response here: https://community.atlassian.com/t5/Bitbucket-questions/Re-How-can-I-use-SSH-in-Bitbucket-Pipelines/qaq-p/764824/comment-id/25670#M25670

Here's a copy of some of it here:

Up to date as of 04/04/2018.

For example. I want to clone repo-2 into a build from repo-1

To use SSH to clone another repository from Bitbucket you need to do the following.

1. In repo-1 generate a Pipelines SSH key to be injected into your build container. Follow Step 1 here: https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html
   • Note that if the known hosts information for bitbucket.org and github.com are injected by default. If you are using any other site you will need to configure Known Hosts, by following Step 2 in the above documentation.
2. Add an Access Key to repo-2. Paste in the public key that was generated in the previous step.
3. Configure your bitbucket-pipelines-yml to use git.

Can you also share some of the links you've looked at previously? I'd like to update them so other people don't run into the same issue as you. I'm in the process of updating some of the highly ranked ones that showed up on Google for me now.

Thanks,

Phil

[EVO_9]

Thank you.

I’m trying to deploy my bitbucket repository to my production server. I’m new to Git but I understand this is the correct way to do this.

as far as I understand, pipelines will deploy the updated files to your production server when a file is updated which sounds perfect. I just can’t get it to work.

[Philip Hodder]

Right. I think all the documentation here may be relevant for you then: https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

If you're trying to deploy files to your production server, you'll probably not be using Git. You'll need to set up an SSH key and then maybe use SFTP or a CLI tool provided by your tools.

How are you currently doing deployments to your production server? What tools do you use? (Going through step by step on how you deploy at the moment would help :) )

[EVO_9]

Thank you.

I followed your link, set up a new ssh key in bitbucket, added it to my remote server, added known_hosts in bitbucket and finally edited the bitbucket-pipelines.yml with the following text:

image: ubuntu:16.04

pipelines:
  default:
    - step:
        - apt-get update -y
        - apt-get install -y git
        - git clone <your_repo_here> # For example, git@bitbucket.org:username/repo-2.git

But it doesn't seem to let me commit the changes?

[Philip Hodder]

Your YAML is slightly incorrect. It should look like this instead:

image: ubuntu:16.04

pipelines:
  default:
    - step:
        script:
          - apt-get update -y
          - apt-get install -y git
          - git clone <your_repo_here> # For example, git@bitbucket.org:username/repo-2.git

You were missing the "script" section. :)

You shouldn't need to do all that git stuff, though. That was for a different specific example.

Now that you've got the ssh key and known hosts set up you should try a simpler version of your initial configuration:

pipelines: 
  default: 
    - step: 
        script:  
          - ssh user@domain 'bash -s' < ./deploy.bash

Have you tried doing a deployment from your own machine. What commands are you using there? You should just be able to copy those commands into the scripts section of your pipeline.

[EVO_9]

Hi Phillip

First of all, thank you for your help.

Hopefully, this makes sense as it's 1 am here and I'm tired.

I can't seem to get my head around this....

I've used the following code:

image: ubuntu:16.04
pipelines:
  default: 
    - step:
      script: 
        - ssh user@domain 'bash -s' < ./deploy.bash

and I get the error:

bash: ./deploy.bash: No such file or directory

I am able to ssh into my remote server / site folder from my computer using the following command on my Macbook Terminal:

ssh -t username@domain -i ~/.ssh/ssh_key "cd ~/site ; bash"

I could just use the following command but if the key is not id_rsa then it won't work and I have to point Terminal to where I've stored the key:

ssh user@domain -i ~/.ssh/ssh_key

But when I try this code in pipelines:

image: ubuntu:16.04

pipelines: 
  default: 
    - step: 
        script:
          - ssh -t user@domaink -i ~/.ssh/ssh_key "cd ~/site ; bash"

I get this error

bash: ssh: command not found

Here's what I am doing from start to finish and maybe you can spot what I'm doing wrong.

• Generate ssh key in Terminal using the following command:

ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/macbook_key

I do this in 2048 and not 4096 just in case.

• I then copy both keys.

• I then log into my sites cPanel / SSH Access / Import keys through the browser

• I then paste the public and private keys in the boxes and authorize the public key

• I name the key 'macbook_key' to match the keys on the mac (I don't add a Passphrase)

• I then check cPanel / File Manager / .ssh : I now have my two ssh keys, I also have an authorize_key and an authorize_key2 file, 4 files in total.

• I can now ssh into my remote server by using the following command:

ssh -t username@domain -i ~/.ssh/ssh_key "cd ~/site ; bash"

What am I missing?

Thanks

[Philip Hodder]

For the:

bash: ssh: command not found

error. You need to install ssh before you can run it. Try adding these commands to your script.

apt-get update -y
apt-get install -y ssh

Lets see where you get after that. If things are behaving then hopefully your build will hang on the bash command. (Watch it and stop the build so you don't use too many build minutes)

If it hangs, then you'll need to add the commands you run into the SSH commands.

If it doesn't work, then it should hopefully be some authentication errors, it which case you'll need to play around and see what's missing auth.

If it's something else. We'll take it from there.

[EVO_9]

Hi Phillip

I've added the 2 lines to the script, see below:

image: ubuntu:16.04

pipelines: 
  default: 
    - step: 
        script:
          - apt-get update -y
          - apt-get install -y ssh
          - ssh -t user@domaink -i ~/.ssh/ssh_key "cd ~/site ; bash"

I'm now getting the following error:

Warning: Identity file /root/.ssh/ssh_key not accessible: No such file or directory.Pseudo-terminal will not be allocated because stdin is not a terminal.Host key verification failed.

On the bright side, it's a new error.

[EVO_9]

Still no further with this...

[Philip Hodder]

Looks like you have a couple errors.

The first one is that the SSH key can't be find. Can you try run SSH without "-i ~/.ssh/ssh_key", as it's not in that location. It should be located in "/root/.ssh". 

The other issue is that you won't be able to have an interactive terminal in Pipelines. (how would you interact with it after you connected?) So you will need to use protocols built on top of SSH (which still handles auth). It depends on what you're trying to do. But if you're doing file upload to another server you will probably need to use SCP or SFTP. If you're doing other stuff then it might be a bit more complicated to operate.

SCP and SFTP starting point: https://unix.stackexchange.com/questions/8707/whats-the-difference-between-sftp-scp-and-fish-protocols
SSH scripts: https://serverfault.com/questions/215756/how-do-i-run-a-local-bash-script-on-remote-machines-via-ssh

The last one is an auth failure. Most likely because the key couldn't be found earlier.

[EVO_9]

Hi Phillip

I have changed the code to the following:

image: ubuntu:16.04

pipelines: 
  default: 
    - step: 
        script:
          - apt-get update -y
          - apt-get install -y ssh
          - ssh -t user@domaink /root/.ssh "cd ~/site ; bash"

But I now get the following error:

Host key verification failed.

If I try the following code:

image: ubuntu:16.04

pipelines: 
  default: 
    - step: 
        script:
          - apt-get update -y
          - apt-get install -y ssh
          - ssh -t user@domaink /root/.ssh/ssh_key "cd ~/site ; bash"

I get this error:

Host key verification failed.

[Philip Hodder]

That means that the known_hosts contains the incorrect information for your site.

Have a look here for a more in depth explanation: https://askubuntu.com/questions/45679/ssh-connection-problem-with-host-key-verification-failed-error

You'll need to check your SSH client is accessing the correct known_hosts file on Pipelines. And that it has the correct fingerprint as your server's domain.

[Michael Born]

You need to install ssh before you can run it.

 @Philip Hodder Is this really true? Why isn't installing SSH mentioned in the SSH docs? https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

[Philip Hodder]

@Michael Born good catch!

Yes you do need it installed. We only add in the SSH keys and known_hosts for you. I've let @Linette (our main documentation writer) know about this. We'll get that added into the docs. Noting it in-product could also be a good idea.

Some Docker images will have SSH installed by default. And others won't. If not, then you'll need to install SSH as part of your script. Or make your own Docker image with SSH installed.

[Linette]

Great call, @Michael Born! I've added a note in the docs that if you are not using the default Docker image, to make sure it's installed or install it. Hopefully that will make people's life much easier!

[UK Print and Promotions Ltd]

@Linette I gave up trying to get this working, I will try again looking at the new docs and see how I get on. I'm new to this process of doing things so if I can get it working then anybody can so I'll be the guinea pig.

[Linette]

You know, if you have a little bit of time up your sleeve, I'd recommend maybe holding off for a little bit, as we have something new coming up soon that should make deployments in general much easier... keep your eye out!

If you are keen to get going now though, I'm sure we'll do all we can to get it working for you!

[Matthew Hailwood]

@Linette is there anymore information on this "something new"?

[Linette]

@Matthew Hailwoodthere's a few moving pieces to get in place, but one of the pieces, 'tasks' is now available in alpha (you can peek here: https://confluence.atlassian.com/bitbucket/tasks-958765631.html). The idea  behind them is to make configuration of your yaml much easier.

We're still increasing the number of tasks available, and we are also working on some other things to try and make the whole SSH experience easier.

If you are not a member of the alpha group you can find out more here: https://confluence.atlassian.com/bitbucket/tasks-958765631.html

[Matthew Hailwood]

Ah thanks @Linette,
Yeah I was looking through those this morning.
None of those tasks really help my usecase unfortunately.

Here's what I need to do for our deployments:

1. npm install
2. npm run prod
3. git pull on remote server
4. run another bash command on the remote server
5. scp specific files (built assets via the npm run prod) to remote server

Any chance you could point me in the right direction?
Let me know if I should create my own question, this just seems appropriate to the original question.

[Linette]

Probably good to create your own question as it's a bit buried at the bottom here, and as it's marked as solved, people won't be necessarily looking at this one... (and it will be super useful for others in a similar situation when yours is solved ;) )

[hwozniak]

I am interested in the answer to this. If it did get posted as a separate question, can someone provide the link?

[Linette]

Hi @hwozniak! Yes, you'll find it here: https://community.atlassian.com/t5/Bitbucket-questions/How-to-deploy-to-remote-server-after-building-assets-in-pipeline/qaq-p/937061?

Hope that helps!

[Fernando Sarmiento]

!WORKS Like charm

image: php:5.6-apache

pipelines:

  default:

   - step:

     script:
       - apt-get update -y

       - apt-get install -y ssh

       - echo $SSH_DEPLOY_KEY > ~/.ssh/id_rsa

       - chmod 600 ~/.ssh/id_rsa

       - ssh -t <user>@<domain.com> -i ~/.ssh/id_rsa /bin/bash -ic 'bash_alias'

bash_alias file content

#!/bin/bash

cd www/test

git pull

 Evidence