How to create a glue job by running the bitbucket pipeline

[srikanth]

How can i create a automated pipeline in bitbucket to push the files from bitbucket repositories to s3 buckets and on that  how can we create a glue job by using the same pipeline by using the files present in s3 bucket.

[Oleksandr Kyrdan]

Hi @srikanth

Thank you for your question!

It's a good case to use Bitbucket Pipes.

Use aws-s3-deploy pipe to deploy files to S3. Add pipe to your bitbucket-pipelines.yaml configuration:

script:
  - pipe: atlassian/aws-s3-deploy:1.1.0
    variables:
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
      AWS_DEFAULT_REGION: 'us-east-1'
      S3_BUCKET: 'my-bucket-name'
      LOCAL_PATH: 'build'
   # install aws cli and run glue commands
   - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.2.9.zip" -o "awscliv2.zip" && unzip awscliv2.zip
   - echo 'c778f4cc55877833679fdd4ae9c94c07d0ac3794d0193da3f18cb14713af615f awscliv2.zip' | sha256sum -c - && ./aws/install
   - aws glue create-job ... your params ...

Best regards,
Oleksandr Kyrdan

[srikanth]

 Thank you for answering to my questions could you please how can i explain me how to achieve this..

For suppose already some glue jobs are already working, in the bitbucket repositories some new code is there which is related to the job, will the pipeline works for to check the list of jobs whether the new was existed or not, if the job was not existed, will it able to create new glue job..

Parallely  will it able update the  code present in hdfs location and same thing like i explained in the above case for glue job is it will able to create new emr job which has to read the code from hdfs location.

Same case for databricks jobs as well.. can you help me how to figure this?