It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to configure pipeline to send prod and staging builds to different aws ec2 with code deploy?

Muhammad Shahid Mar 13, 2018

Hi Team,

We have two ec2 instances one is for production and 2nd one is for development/staging.  Similarly we have two branches, master and develop. Master's build is in used at production ec2 instance and develop is for staging server.

I have configured bitbucket-pipeline.yml with branches flow.  But how I can control appsec.yml or any other feature to manage flow of builds to copy over just to relevant ec2 instances.   At AWS codedeploy I can configure deployment group and when there would be commit either at master or develop branch, that build will be pushed to codedeploy deployment process and it will push code to deployment group where both servers are in same deployment group.

So question is how and where we can control build deployment to respective instance only with feature bitbucket to aws codedeploy.

2 answers

0 votes
Billy Poggi Community Leader Mar 14, 2018

Hi Muhammad, 

I was reading up on your issue and I wanted to learn more about the situation.  In doing so, I found this page.

https://hackernoon.com/deploy-to-ec2-with-aws-codedeploy-from-bitbucket-pipelines-4f403e96d50c

 

How do you currently select which system to deploy to? dev/ staging

-Billy

Muhammad Shahid Mar 14, 2018 • edited

Thank you Billy, I have check out this post and my setup is 100% relevant to this post.  This setup is good for one deployment group. For example in DG1 (CodeDeploy) group if I put both servers production and staging and when pipelines will run and push code/build at that point how should I control deployment to just staging or just to production?

This is my bitbucker-pipeline.yml

pipelines: branches: develop: - step: name: Run Build and Deploy to Staging Server image: node:6 script: npm install - npm run build  - codedeploy_deploy.py # run the deployment script  branches: master: - step: name: Run Build and Deploy to Prod Server image: node:6 script: - npm install - npm run build  - python codedeploy_deploy.py

appsec.yml

version: 0.0os: linuxfiles: - source: / destination: /var/www/apphooks: BeforeInstall: - location: scripts/install_dependencies timeout: 300 runas: root - location: scripts/start_server timeout: 300 runas: root ApplicationStop: - location: scripts/stop_server timeout: 300 runas: root
Jeroen De Raedt Atlassian Team Mar 15, 2018

Am I correct in assuming that the real difference is that you want to use different appsec.yml files for staging and production? 

If that is the case, 1 solution could be to create 2 directories in your repository (/staging and /production) and in each you store the appropriate version of the appsec.yml file. When doing deployments, you change directories to the appropriate folder and then execute the codedeploy script. 

Muhammad Shahid Mar 15, 2018

Thanks for reply, basically issue is not handle appsec.yml.  We can have similar path for staging and production server.  Issue is at AWS deployment group.  There we can select deployment group with bunch of ec2 instances.  But how we can deploy to just staging and production instances separately.  At pipeline we can define environmental variables but for singal deployment group. So how to deploy to staging and just for production deployment groups.

0 votes
Muhammad Shahid Mar 15, 2018 • edited
.
Jeroen De Raedt Atlassian Team Mar 15, 2018

Ah right I see. 

I suspect that the python script you use has hardcoded in it to use a certain environment variable to configure the AWS deployment group? 

In the step that does a deployment, you could add a command to your script in bitbucket-pipelines.yml that overwrites the value of your deployment group variable.

 

For example: 

pipelines: 
branches:
develop:
- step:
name: Run Build and Deploy to Staging Server
image: node:6
script:
- npm install
- npm run build
- export DEPLOYMENT_GROUP="STAGING"
- codedeploy_deploy.py # run the deployment script
master:
- step:
name: Run Build and Deploy to Prod Server
image: node:6
script:
- npm install
- npm run build
- export DEPLOYMENT_GROUP="PRODUCTION"
- python codedeploy_deploy.py
Billy Poggi Community Leader Mar 17, 2018

Jeroen, 

It looks like this will deploy to both groups at the same time.  Is that correct?

Muhammad Shahid Mar 17, 2018 • edited

No, That can be controller through bitbucket-pipeline.yml branch flow. Actual thing was at code deploy deployment and now if we can over ride environment variables through deployment script that would be super cool. Will give it a try. @Jeroen De Raedt can you please share any possible deployment script?  or edit this one :

https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/73b7c31b0a72a038ea0a9b46e457392c45ce76da/codedeploy_deploy.py?at=master&fileviewer=file-view-default

Redux Mar 29, 2018

Muhammad Shahid / Billy Poggi /Jeroen De Raedt

Could you please share the current status on this. Will this  export command overrides Deployment group environment variables?

Do we need to change anything on the python script. I'm stuck here on this. Kindly advise. 

Thanks in advance

Muhammad Shahid Mar 30, 2018

@ReduxNot yet, busy with other tasks and still not sure variables of python script may override environmental variables of pipelines.  Please advise if you go through with it. Thanks

Aswin Satish Oct 11, 2018

Hi all,

 

We are following this same procedure for deployment, except that the deployment goes to same server, but to different paths in 2 appsec.yml, one for master and other development branch. The issue is that when commit occurs in any branch, the files in other branch are getting removed. For eg: if we commit to master, all the files that are already in development branch get deleted.

Can anyone suggest a solution? 

Aswin Satish Oct 15, 2018

Any one, please

Aswin Satish Oct 15, 2018

@Muhammad Shahid

Do you have any ideas regarding my issue?

Graham Gatus Atlassian Team Oct 17, 2018

@Aswin Satish One solution is to use 2 appspec files, one per branch:

appspec-master.yml:

version: 0.0
os: linux

files:
- source: ./
destination: /opt/master-path

appspec-develop.yml:

version: 0.0
os: linux

files:
- source: ./
destination: /opt/develop-path

 

In your pipelines config, depending on the branch, you could assemble your CodeDeploy artifact using either file Appspec file, e.g

 

pipelines:
default:
- step:
script:
- run-tests.sh
branches:
master:
- step:
script:
- mkdir application
- mv appspec-master.yml application/appspec.yml
- <add other files to the application directory>
- >
aws deploy push --application-name <your app> \
--source application \
--s3-location s3://<your bucket>/<revision key>
develop:
- step:
script:
- mkdir application
- mv appspec-master.yml application/appspec.yml
- <add other files to the application directory>
- >
aws deploy push --application-name <your app> \
--source application \
--s3-location s3://<your bucket>/<revision key>

There is a $BITBUCKET_BRANCH environment variable that you could also use to generate a appspec.yml file dynamically. You could use it to generate an appspec.yml file where the files destination is the branch name. That way, each branch would get deployed to its own path on the EC2 instance.

Aswin Satish Nov 05, 2018

@Graham Gatus Thanks for your reply. 

I'm using different appspec files for master and production branches:

Master:

version: 0.0
os: linux
files:
-
source: /
destination: /home/ec2-user/

Production:

version: 0.0
os: linux
files:
-
source: /
destination: /var/www/

My pipeline file for master:

pipelines: 
branches:
master:
- step:
image: python:3.5.1
script:
-
apt-get update # required to install zip
-
apt-get install -y zip # required for packaging up the application
-
pip install boto3==1.3.0 # required for codedeploy_deploy.py
-
zip -r /tmp/artifact.zip * # package up the application for deployment
-
python codedeploy_deploy.py # run the deployment script

For production:

pipelines: 
branches:
production:
- step:
image: python:3.5.1
script:
-
apt-get update # required to install zip
-
apt-get install -y zip # required for packaging up the application
-
pip install boto3==1.3.0 # required for codedeploy_deploy.py
-
zip -r /tmp/artifact.zip * # package up the application for deployment
-
python codedeploy_deploy.py # run the deployment script

And I'm using a codedeploy file which contains the following variables:

APPLICATION_NAME: Name of codedeploy application

S3_BUCKET: Name of s3 bucket

DEPLOYMENT_GROUP_NAME: Codedeploy group name

DEPLOYMENT_CONFIG: CodeDeployDefault.OneAtATime

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

AWS_ACCESS_KEY_ID

 

I need my code in bitbucket to be copied to ​/home/ec2-user when any push or merge happens in master branch and to /var/www/ when any push or merge happens in production branch. 

Everything is working fine, however after a push or merge in one branch, on another push to other branch overwrite files in first branch

Aswin Satish Nov 12, 2018

Any ideas ?

Aswin Satish Nov 21, 2018

Still waiting for a reply..Can anyone help, please

Tom Champion I'm New Here Apr 12, 2019

@Aswin SatishI know this is more than a little late, but if you're deploying a staging/production version of your app to the the same EC2 instance, in order for CodeDeploy to not delete the previously deployed app (lets say develop) you should create two CodeDeploy applications in the AWS console and then define the application_name and deployment_group in your bitbucket-piplines.yaml file for each branch.

 

Hopefully this is useful for anyone else who visits this page in 2019.. It worked for me and I'm now able to deploy to both staging/prod on one EC2 instance.

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Posted in Off-topic

What podcasts are you listening to right now?

Like the title says: what podcasts are in your rotation right now? Here are some favorites I've been in to from the last few weeks. The Ringer NBA Show I've been diving deep into the NBA Playoffs...

814 views 16 9
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you