OIDC not working with aws-cli but works with pipes

[Deb Mohan]

I am trying to use OIDC with a role and a web identity on aws. I followed the steps from https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/

As noted at the end of the post, the OIDC works when I use atlassian pipes but it does not work with the export and aws cli options.

# bitbucket pipeline - this one fails
image: amazon/aws-cli
pipelines:
default:
 - step:
name: Connect to AWS using OIDC
oidc: true
script:
 - unset AWS_SECRET_ACCESS_KEY
 - unset AWS_ACCESS_KEY_ID
 - export AWS_REGION=$AWS_REGION
 - export AWS_ROLE_ARN=arn:aws:iam::1234567890:role/MyRole
 - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
 - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
 - printenv BITBUCKET_STEP_OIDC_TOKEN
 - printenv AWS_REGION
 - printenv AWS_ROLE_ARN
 - aws sts assume-role-with-web-identity --role-arn arn:aws:iam::1234567890:role/MyRole --role-session-name build-session --web-identity-token "$BITBUCKET_STEP_OIDC_TOKEN" --duration-seconds 1000
 - aws sts get-caller-identity

# Output from the above:

printenv AWS_REGION
us-east-2

printenv AWS_ROLE_ARN
arn:aws:iam::1234567890:role/MyRole

printenv BITBUCKET_STEP_OIDC_TOKEN
<nothing here>

# Error:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

# this one works with the same AccessIdentityRole to copy files from my build to the s3 bucket

- pipe: atlassian/aws-s3-deploy:1.1.0
  variables:
    AWS_DEFAULT_REGION: $AWS_REGION # Optional if already defined in the context or OIDC used.
    AWS_OIDC_ROLE_ARN: $AWS_OIDC_ROLE_ARN # Optional by default. Required for OpenID Connect (OIDC) authentication.
    S3_BUCKET: mygreat-bucket
    LOCAL_PATH: 'build'
    CACHE_CONTROL: 'max-age=86400'

# Role Policy:
 {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": "*",
       "Resource": "*"
     }

   ]
 }

# Trust Relationship:

 {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Principal": {
          "Federated": "arn:aws:iam::1234567890:oidc-provider/api.bitbucket.org/2.0/workspaces/myworkspace/pipelines-config/identity/oidc"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
        "StringEquals": {
          "api.bitbucket.org/2.0/workspaces/myworkspace/pipelines-config/identity/oidc:sub": "w234y098-f0tk-823a-45m1-qwrth8912n00:*"
         }
       }
     }
   ]
 }

[Patrik S]

Hello @Deb Mohan ,

Thank you for reaching out to Atlassian Support.

From the error message, you are receiving it seems the policy/trust relationship you are currently using is not allowing the pipeline's environment to assume the role :

Accordingly to the trust relationship you have shared, it seems you have configured it to allow only the repository with the UUID below to assume the role :

w234y098-f0tk-823a-45m1-qwrth8912n00

Checking for that UUID internally I was not able to find it associated with any repository. Could you please double-check if the value is correct? You can do that by following the below steps : 

• Access the repository where the pipeline runs
• Go to Repository Settings
• Click on OpenID Connect under Pipelines
• Copy the content of the field Repository UUID
• Paste it in your trust relationship rule, including the braces {}, as below : 

  "StringEquals": {
            "api.bitbucket.org/2.0/workspaces/myworkspace/pipelines-config/identity/oidc:sub": "{REPOSITORY_UUID}:*"

  You can also refer to the following document for instructions on how you can add conditions to your trust relationship in AWS : 
  https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/#Allowing-only-a-specific-repository-to-assume-the-role

If that does not work, you can also try to remove the trust relationship condition temporarily and test running your build again to check if the command you are executing is able to assume the role.

Hope that helps, let me know in case you have any questions!

Thank you, @Deb Mohan .

Kind regards,

Patrik S

[Deb Mohan]

@Patrik S

I am using a repository ID. for the purpose of security I did not share the repository id here as it is a public domain.

If you provide me with an email I can share my repo ID.

I have already tested the the trust with the braces and it did not work. Currently the only way the OIDC is working is with the audience.

[Patrik S]

Hello @Deb Mohan ,

As I suspect the issue is the string not matching your trust relationship, could you try removing the "String equals" section of your trust relationship, leaving the "Conditions" object empty, and try again? 

You can also print the value of the variable inside your build

echo $BITBUCKET_STEP_OIDC_TOKEN

so you will get the token bitbucket will use to assume the role in AWS. You can then debug that token in jwt.io and see if the "sub" field of the JSON matches the Repository UUID value you have configured in the trust relationship. The sub claim is formed like the following : 

{REPOSITORY_UUID}[:{ENVIRONMENT_UUID}]:{STEP_UUID}
E.g. 
{1de489be-ce6a-42a0-a8c8-eadbf1174ac7}:{bd715740-c970-486b-b68a-b421ec2a1f8b}:{759de0c6-eaee-4eaa-b7a6-c507eec759a7} 
{1de489be-ce6a-42a0-a8c8-eadbf1174ac7}:{759de0c6-eaee-4eaa-b7a6-c507eec759a7}

ENVIRONMENT_UUID: This part shows up only if the step is assigned to a deployment environment. 

Kind regards,

Patrik S

[Deb Mohan]

Hi @Patrik S ,

I have been using Trust Relationship with Audience and it works and it did without any conditions too. The problem is with the sub option. Looks like something is wrong with the way the policy is created or the string is passed. I checked the cloudtrail logs and do not see anything wrong. I am not sure what the problem would be.

As I mentioned earlier for some reason 

echo $BITBUCKET_STEP_OIDC_TOKEN

does not work. I get nothing in the response.

[Patrik S]

Hello @Deb Mohan ,

You are not able to see the value of the OIDC Token because the $BITBUCKET_STEP_OIDC_TOKEN is considered a secure variable, so its value is masked from the logs for security reasons.

In order to check the value of that variable you can output it to a file, and make that file available as an artifact, as the below example :

- step:
name: 'test'
oidc: true
script:
 - echo $BITBUCKET_STEP_OIDC_TOKEN > my_token.txt
artifacts:
 - my_token.txt

After the build finishes, the file should be available in the artifacts tab of that build, so you can download and verify the value of the token, and inspect it in jwt.io to check for the "sub" value.

Kind regards,

Patrik S

[Deb Mohan]

@Patrik S

Thank you for the tip. I verified the sub and I have it correct in my Trust Relationship policy. Still does not work.

[Patrik S]

Hello @Deb Mohan ,

In this case, I would suggest trying to enable the Cloud Trail logs on AWS for IAM and STS services, so you can potentially get more information on why AWS is denying your pipeline to assume the role when using the sub filter.

I found the following aws documentation on how to enable the logs for the calls to IAM and STS APIs that might be of help : 

• https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html

Thank you, @Deb Mohan .

Kind regards,

Patrik S

[blake.dobay]

Can confirm, this is definitely an issue. Audience works fine but sub is not functioning. Have confirmed via the extracted token that the repo UUID is correct compared to my AWS policy. 

Inspection of  Cloudtrail only gives: "errorCode": "AccessDenied",
"errorMessage": "An unknown error occurred"

[Patrik S]

Hello @blake.dobay ,

Welcome to Atlassian Community

Could you try setting up the Trust Relationship in AWS using the condition StringLike , instead of StringEquals, like the below example : 

{

  "Effect": "Allow",

  "Principal": {

    "Federated": "arn:aws:iam::{AWS_ACCOUNT_NUMBER}:oidc-provider/api.bitbucket.org/2.0/workspaces/{WORKSPACE}/pipelines-config/identity/oidc"

  },

  "Action": "sts:AssumeRoleWithWebIdentity",

  "Condition": {

    "StringLike": {

      "api.bitbucket.org/2.0/workspaces/{WORKSPACE}/pipelines-config/identity/oidc:sub": "{REPO_UUID}:*"

    }

  }

}

Reference: https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/#Allowing-only-a-specific-repository-to-assume-the-role

Thank you,
Patrik S

[Deb Mohan]

Tried with no condition and it works as expected. I have been using the audience and it works too. Just that when I want to use the repositoryUuid with the sub, the build breaks. I guess there is a problem with the sub.

Also as suggested, I am not getting any response for the echo statement.