How to configure pipeline to send prod and staging builds to different aws ec2 with code deploy?

Hi Team,

We have two ec2 instances one is for production and 2nd one is for development/staging.  Similarly we have two branches, master and develop. Master's build is in used at production ec2 instance and develop is for staging server.

I have configured bitbucket-pipeline.yml with branches flow.  But how I can control appsec.yml or any other feature to manage flow of builds to copy over just to relevant ec2 instances.   At AWS codedeploy I can configure deployment group and when there would be commit either at master or develop branch, that build will be pushed to codedeploy deployment process and it will push code to deployment group where both servers are in same deployment group.

So question is how and where we can control build deployment to respective instance only with feature bitbucket to aws codedeploy.

2 answers

Hi Muhammad, 

I was reading up on your issue and I wanted to learn more about the situation.  In doing so, I found this page.

https://hackernoon.com/deploy-to-ec2-with-aws-codedeploy-from-bitbucket-pipelines-4f403e96d50c

 

How do you currently select which system to deploy to? dev/ staging

-Billy

Thank you Billy, I have check out this post and my setup is 100% relevant to this post.  This setup is good for one deployment group. For example in DG1 (CodeDeploy) group if I put both servers production and staging and when pipelines will run and push code/build at that point how should I control deployment to just staging or just to production?

This is my bitbucker-pipeline.yml

pipelines: branches: develop: - step: name: Run Build and Deploy to Staging Server image: node:6 script: npm install - npm run build  - codedeploy_deploy.py # run the deployment script  branches: master: - step: name: Run Build and Deploy to Prod Server image: node:6 script: - npm install - npm run build  - python codedeploy_deploy.py

appsec.yml

version: 0.0os: linuxfiles: - source: / destination: /var/www/apphooks: BeforeInstall: - location: scripts/install_dependencies timeout: 300 runas: root - location: scripts/start_server timeout: 300 runas: root ApplicationStop: - location: scripts/stop_server timeout: 300 runas: root

Am I correct in assuming that the real difference is that you want to use different appsec.yml files for staging and production? 

If that is the case, 1 solution could be to create 2 directories in your repository (/staging and /production) and in each you store the appropriate version of the appsec.yml file. When doing deployments, you change directories to the appropriate folder and then execute the codedeploy script. 

Thanks for reply, basically issue is not handle appsec.yml.  We can have similar path for staging and production server.  Issue is at AWS deployment group.  There we can select deployment group with bunch of ec2 instances.  But how we can deploy to just staging and production instances separately.  At pipeline we can define environmental variables but for singal deployment group. So how to deploy to staging and just for production deployment groups.

Ah right I see. 

I suspect that the python script you use has hardcoded in it to use a certain environment variable to configure the AWS deployment group? 

In the step that does a deployment, you could add a command to your script in bitbucket-pipelines.yml that overwrites the value of your deployment group variable.

 

For example: 

pipelines: 
branches:
develop:
- step:
name: Run Build and Deploy to Staging Server
image: node:6
script:
- npm install
- npm run build
- export DEPLOYMENT_GROUP="STAGING"
- codedeploy_deploy.py # run the deployment script
master:
- step:
name: Run Build and Deploy to Prod Server
image: node:6
script:
- npm install
- npm run build
- export DEPLOYMENT_GROUP="PRODUCTION"
- python codedeploy_deploy.py

Jeroen, 

It looks like this will deploy to both groups at the same time.  Is that correct?

No, That can be controller through bitbucket-pipeline.yml branch flow. Actual thing was at code deploy deployment and now if we can over ride environment variables through deployment script that would be super cool. Will give it a try. @Jeroen De Raedt can you please share any possible deployment script?  or edit this one :

https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/73b7c31b0a72a038ea0a9b46e457392c45ce76da/codedeploy_deploy.py?at=master&fileviewer=file-view-default

Muhammad Shahid / Billy Poggi /Jeroen De Raedt

Could you please share the current status on this. Will this  export command overrides Deployment group environment variables?

Do we need to change anything on the python script. I'm stuck here on this. Kindly advise. 

Thanks in advance

@ReduxNot yet, busy with other tasks and still not sure variables of python script may override environmental variables of pipelines.  Please advise if you go through with it. Thanks

Hi all,

 

We are following this same procedure for deployment, except that the deployment goes to same server, but to different paths in 2 appsec.yml, one for master and other development branch. The issue is that when commit occurs in any branch, the files in other branch are getting removed. For eg: if we commit to master, all the files that are already in development branch get deleted.

Can anyone suggest a solution? 

Any one, please

@Muhammad Shahid

Do you have any ideas regarding my issue?

Graham Gatus Atlassian Team Oct 17, 2018

@Aswin Satish One solution is to use 2 appspec files, one per branch:

appspec-master.yml:

version: 0.0
os: linux

files:
- source: ./
destination: /opt/master-path

appspec-develop.yml:

version: 0.0
os: linux

files:
- source: ./
destination: /opt/develop-path

 

In your pipelines config, depending on the branch, you could assemble your CodeDeploy artifact using either file Appspec file, e.g

 

pipelines:
default:
- step:
script:
- run-tests.sh
branches:
master:
- step:
script:
- mkdir application
- mv appspec-master.yml application/appspec.yml
- <add other files to the application directory>
- >
aws deploy push --application-name <your app> \
--source application \
--s3-location s3://<your bucket>/<revision key>
develop:
- step:
script:
- mkdir application
- mv appspec-master.yml application/appspec.yml
- <add other files to the application directory>
- >
aws deploy push --application-name <your app> \
--source application \
--s3-location s3://<your bucket>/<revision key>

There is a $BITBUCKET_BRANCH environment variable that you could also use to generate a appspec.yml file dynamically. You could use it to generate an appspec.yml file where the files destination is the branch name. That way, each branch would get deployed to its own path on the EC2 instance.

@Graham Gatus Thanks for your reply. 

I'm using different appspec files for master and production branches:

Master:

version: 0.0
os: linux
files:
-
source: /
destination: /home/ec2-user/

Production:

version: 0.0
os: linux
files:
-
source: /
destination: /var/www/

My pipeline file for master:

pipelines: 
branches:
master:
- step:
image: python:3.5.1
script:
-
apt-get update # required to install zip
-
apt-get install -y zip # required for packaging up the application
-
pip install boto3==1.3.0 # required for codedeploy_deploy.py
-
zip -r /tmp/artifact.zip * # package up the application for deployment
-
python codedeploy_deploy.py # run the deployment script

For production:

pipelines: 
branches:
production:
- step:
image: python:3.5.1
script:
-
apt-get update # required to install zip
-
apt-get install -y zip # required for packaging up the application
-
pip install boto3==1.3.0 # required for codedeploy_deploy.py
-
zip -r /tmp/artifact.zip * # package up the application for deployment
-
python codedeploy_deploy.py # run the deployment script

And I'm using a codedeploy file which contains the following variables:

APPLICATION_NAME: Name of codedeploy application

S3_BUCKET: Name of s3 bucket

DEPLOYMENT_GROUP_NAME: Codedeploy group name

DEPLOYMENT_CONFIG: CodeDeployDefault.OneAtATime

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

AWS_ACCESS_KEY_ID

 

I need my code in bitbucket to be copied to ​/home/ec2-user when any push or merge happens in master branch and to /var/www/ when any push or merge happens in production branch. 

Everything is working fine, however after a push or merge in one branch, on another push to other branch overwrite files in first branch

Still waiting for a reply..Can anyone help, please

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Thursday in Agile

How Scrum works? It starts with training and education

To answer “How scrum works,” most of the teams I've worked with first addressed the question: “where to start?”  That question applies to both implementation and improvements on the Scrum framew...

180 views 3 5
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you