How to use OpenID Connect with multiple AWS accounts

[Aaron Durant]

I have multiple AWS accounts for different stages (i.e. One for development, one for staging, and one for production)

In each account, I have created an OpenID Connect Identity Provider, using the URL and audience found in my Organisations BitBucket account, and an associated IAM Role. There are the same in every AWS account, except the name of the IAM Role changes to reflect the stage.

I have a BitBucket pipeline in a repository to deploy the application to AWS. I have set Deployment variables in the repository settings for `CDK_DEFAULT_ACCOUNT` and `AWS_ROLE_ARN` to reflect the different accounts I want to deploy to for each stage.

branches:
  staging:
    - stage:
        name: Build and Deploy Staging
        deployment: Staging
        steps:
          - step: *install_dependencies
          - step: *test
          - step: *build
          - step: *deploy
  development:
    - stage:
        name: Build and Deploy Development
        deployment: Development
        steps:
          - step: *install_dependencies
          - step: *test
          - step: *build
          - step: *deploy

When I run the pipeline on the `development` branch, it runs correctly and deploys to AWS.
When I run the pipeline on the `staging` branch, I get

Building Assets Failed: Error: Need to perform AWS calls for account XXXXXXXXXX, but no credentials have been configured

despite having the same variables defined.

Is there a limitation on OpenID Connect and multiple AWS accounts, or some extra configuration needed? Or simply something set up incorrectly in my pipeline?

[Aaron Durant]

I've managed to solve the problem - the issue was that the IAM Role associated with the OpenID Connect Provider in the staging account had permission for `sts:AssumeRole`, instead of the required `sts:AssumeRoleWithWebIdentity`

I added -vv and --debug flags to the cdk command, which provided more details on the authentication error

[Igor Stoyanov]

@Aaron Durant  Great! You could mark your answer as Accepted, so the other users could see that your case has solution.

Regards, Igor

[Igor Stoyanov]

@Aaron Durant  hi. Check this guide.

 - step:
     oidc: true
     script:
       - export AWS_REGION=us-west-2
       - export AWS_ROLE_ARN=arn:aws:iam::XXXXXXXXXXXX:role/oidc-demo
       - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
       - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
       - aws s3 cp s3://bucket/XXXXXX ./XXXXXXX

Regards, Igor

[Aaron Durant]

Sorry, I should have posted my deploy step as well

- step: &deploy
    name: Deploy
    caches:
      - node
    oidc: true
    script:
      - npm install
      - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
      - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
      - npx cdk deploy

`AWS_REGION` and `AWS_ROLE_ARN` are defined in pipeline/deployment variables. Do I have to explicitly export them inside the script?

[Igor Stoyanov]

@Aaron Durant  variables defined in pipeline/deployment variables should be available without export. You could test this case with `echo <any unsecured variable>` command.

Check the page for bitbucket variables and secrets

Regards, Igor

[Aaron Durant]

Thanks Igor. 
I have printed all the variables involved for each Deployment, and verified they are correct, but the AWS credentials are still not recognised for the second account.

Is there anything else I can try?

Thank you

[Igor Stoyanov]

@Aaron Durant hi. Please check the same for `staging` branch, where you investigated the error.

Regards, Igor

[Igor Stoyanov]

Sorry, i read Development instead of Deployment.

[Aaron Durant]

I have verified the variables on both the `staging` and `development` branches, and in both cases they are correct

[Igor Stoyanov]

@Aaron Durant  i think you should dive into this:
https://github.com/aws/aws-cdk/issues/20935,

especially:

the CDK CLI does not have the capability to automatically pick the correct profile for the configured stack environment. The CLI needs some set of credentials to start with, and if you do not provide the --profile then it will assume that you are using the default profile.

If you have bootstrapped your accounts then there will be a set of IAM roles that have been created in those accounts. When you run a CDK command, the CLI will use your configured credentials to assume the bootstrapped roles in each account.

So for example if you run npx cdk synth --profile=dev --all it will use the credentials from your dev profile and will first assume a specific role in the target account.