Using OpenID Connect to pull private images from ECR

[Luiz Rocha]

Pipeline fails when using OpenID Connect functionality to pull private images from ECR

• OIDC role has access to ECR

[mkleint · Accepted solution]

Can you share your IAM role's trust relationship definition? Access denied is usually returned by ECR when the conditions in trust relations are not met

[Luiz Rocha]

{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::{AWS_ACCOUNT_NUMBER}:oidc-provider/api.bitbucket.org/2.0/workspaces/{WORKSPACE}/pipelines-config/identity/oidc" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringLike": { "api.bitbucket.org/2.0/workspaces/{WORKSPACE}/pipelines-config/identity/oidc:sub": "*:*" } } }

Hi, above is the IAM role trust relationship definitions. This is the instructions I followed: https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/ 

[mkleint]

oh, it looks like we are referencing the wrong documentation from https://support.atlassian.com/bitbucket-cloud/docs/use-docker-images-as-build-environments/#Private-images-hosted-by-AWS-ECR--EC2-Container-Registry-

it should point to https://support.atlassian.com/bitbucket-cloud/docs/use-aws-ecr-images-in-pipelines-with-openid-connect/ where the relevant IAM role conditions actually allow pulling ECR images

[Luiz Rocha]

Yes, but I shared the IAM role trust relationship. The IAM permissions already have the required permissions as the documentation as well. I added a wild card to check if there was any missed permissions however the issue still happening. 

 {
    "Sid": "",
    "Effect": "Allow",
    "Action": [
    "sts:Get*"
    ],
    "Resource": "*"
},
{
    "Sid": "",
    "Effect": "Allow",
    "Action": "ecr:*",
    "Resource": "*"
} 

[mkleint]

ok,

just to be sure, you actually have the real account number in place of

{AWS_ACCOUNT_NUMBER}

and real workspace name in place of

{WORKSPACE}

?

what is the oidc url and audience you are using?

in our tests we have something along the lines of

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::99999999999999:oidc-provider/api.bitbucket.org/2.0/workspaces/XXXXXXXX/pipelines-config/identity/oidc"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"api.bitbucket.org/2.0/workspaces/XXXXXXXX/pipelines-config/identity/oidc:aud": "ari:cloud:bitbucket::workspace/fffffffff-ffff-ffff-fffff-ffffffff"
}
}
}
]
}

I suppose the StringLike + "*:* should work fine too. Maybe first try with something similar to what we have in the docs and start tweaking it step by step.

[Luiz Rocha]

Thanks for your answers @mkleint . It seems that there is an issue when I add the condition to allow only requests from Bitbucket Pipelines IP range to assume the role because when I remove the IP conditions it works. How can I add both of the conditions without getting ECR denied? Here is the documentation I followed to add the Bitbucket IP range restriction https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/ 

{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::{AWS_ACCOUNT_NUMBER}:oidc-provider/api.bitbucket.org/2.0/workspaces/{WORKSPACE}/pipelines-config/identity/oidc" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringLike": { "api.bitbucket.org/2.0/workspaces/{WORKSPACE}/pipelines-config/identity/oidc:sub": "{REPO_UUID}:*" }, "IpAddress": { "aws:SourceIp": [ "34.199.54.113/32", "34.232.25.90/32", "34.232.119.183/32", "34.236.25.177/32", "35.171.175.212/32", "52.54.90.98/32", "52.202.195.162/32", "52.203.14.55/32", "52.204.96.37/32", "34.218.156.209/32", "34.218.168.212/32", "52.41.219.63/32", "35.155.178.254/32", "35.160.177.10/32", "34.216.18.129/32" ] } } }

[mkleint]

Yeah, for ECR images, your IP ranges will be different from the general pipelines usecase. The ECR login dance is done outside of the kubernetes cluster in our services as we need to pass the correct login token to the kubernetes pod definition already. Please check https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/ (last section) on the IP ranges you need to add for the ECR usecase. It could be that you will need to keep both ranges in your definition as the access is split between the service and the kubernetes cluster.

[Luiz Rocha]

Added ip range from the last section but still not working

 "104.192.136.0/21", "13.236.8.128/25", "13.52.5.0/25", "18.136.214.0/25", "18.184.99.128/25", "18.205.93.0/27", "18.234.32.128/25", "18.246.31.128/25", "185.166.140.0/22", "2401:1d80:3000::/36", "2406:da18:809:e00::/56", "2406:da1c:1e0:a200::/56", "2600:1f14:824:300::/56", "2600:1f18:2146:e300::/56", "2600:1f1c:cc5:2300::/56", "2a05:d014:f99:dd00::/56", "2a05:d018:34d:5800::/56", "34.199.54.113/32", "34.216.18.129/32", "34.218.156.209/32", "34.218.168.212/32", "34.232.119.183/32", "34.232.25.90/32", "34.236.25.177/32", "35.155.178.254/32", "35.160.177.10/32", "35.171.175.212/32", "52.202.195.162/32", "52.203.14.55/32", "52.204.96.37/32", "52.215.192.128/25", "52.41.219.63/32", "52.54.90.98/32"

[mkleint]

I suppose you will have to debug that from cloud trail. Not sure if what you pasted is the whole list but I meant this bit from the document (see screenshot)

[Luiz Rocha]

I got all IPs from the "Atlassian cloud IP ranges for AWS ECR" and added to the IP ranges for general pipeline use cases.

[mkleint]

I'm investigating this. it indeed looks like the source IP is not from the ranges declared.

[mkleint]

@Luiz Rocha I've found the problem. Due to a coding mistake we were routing some calls via the public IPs (ecr login itself) but  some were omitted (sts assume role with web identity). I have a fix ready but due to unrelated circumstances can't deploy right now. Should be fixed by the edn of the week. I'll let ya know. Thanks for reporting the problem.

[Luiz Rocha]

Good news! Thanks for you support @mkleint 

[mkleint]

The patch has been deployed and the following ip range definition worked for me. (as per documentation)

 "IpAddress": {
"aws:SourceIp": [
"34.199.54.113/32",
"34.232.25.90/32",
"34.232.119.183/32",
"34.236.25.177/32",
"35.171.175.212/32",
"52.54.90.98/32",
"52.202.195.162/32",
"52.203.14.55/32",
"52.204.96.37/32",
"34.218.156.209/32",
"34.218.168.212/32",
"52.41.219.63/32",
"35.155.178.254/32",
"35.160.177.10/32",
"34.216.18.129/32",
"52.41.219.63/32",
"34.216.18.129/32",
"13.236.8.128/25",
"18.246.31.128/25",
"34.236.25.177/32",
"185.166.140.0/22",
"34.199.54.113/32",
"35.155.178.254/32",
"52.204.96.37/32",
"35.160.177.10/32",
"52.203.14.55/32",
"18.184.99.128/25",
"52.215.192.128/25",
"104.192.136.0/21",
"18.205.93.0/27",
"35.171.175.212/32",
"18.136.214.0/25",
"52.202.195.162/32",
"13.52.5.0/25",
"34.218.168.212/32",
"18.234.32.128/25",
"34.218.156.209/32",
"52.54.90.98/32",
"34.232.119.183/32",
"34.232.25.90/32"
]
}

[Luiz Rocha]

Yesss, it works now! Fantastic job and answers @mkleint. Thank you!

[Fernando Lapa]

@mkleint please, can you update the documentation in this link with this information?

https://support.atlassian.com/bitbucket-cloud/docs/use-aws-ecr-images-in-pipelines-with-openid-connect/

[Halyna Berezovska]

@Luiz Rocha thanks for raising the issue. If you have some related logs, please share that, it will help to discover root cause.

Cheers, Galyna

[Luiz Rocha]