Failed to retrieve temporary AWS credentials: com.amazonaws.SdkClientException

Question

Since last few days we have been getting below errors in Bamboo build jobs while performing AWS operations for which we are using Identity Federation for AWS - Connector to fetch the temporary credentials for specific IAM user .

Somehow, It works if we give build after 5-10 minutes.

It was working fine since last week. We have checked that IAM users access key status is Active. Got to know from AWS side that it could be because of IAM throttling due to huge API calls like 10 per sec but still it is not confirmed root cause.

"Failed to retrieve temporary AWS credentials: com.amazonaws.SdkClientException: Unable to execute HTTP request: Remote host closed connection during handshake"

Steffen Opel [Utoolity] · Answer

Hi Akshay,

Thanks for signing up, I've just made you the reporter of the support request created from your post so we can collaborate on solving this puzzle over there, just quickly:

* You are using the most current releases of both apps, so no updates are needed right now.
* An expired license should not matter, because Identity Federation for AWS is free for licensees of other Utoolity apps that integrate them: This works automatically, i.e. Identity Federation for AWS will detect other licensed Utoolity apps and ignore its own license status then – you can simply ignore or remove the expired license, and I'll also provide you a complimentary license with a 100% discount code via the support request to avoid future confusion around this usability issue (unfortunately Atlassian does not support cross-app licenses).
* The referenced issue UAA-29 likely has an unrelated root cause, see below.

One more thing to ask where to set these variable in bamboo. I tried to set them as Global variable and given a build but that shows default maxErrorRetry as 7.

Unfortunately our documentation erroneously referred to the variables with the bamboo.* prefix required for using variables within the build, which must not be used when defining variables, sorry you had to discover this the hard way! I've updated my answer with the fix to hopefully prevent others from running into this easy to trip over detail.

Fortunately this means the proposed workaround might still solve the intermittent temporary credentials errors, so let's continue the conversation via the support request.

Cheers,
Steffen

Akshay Modi · Answer

Hi Steffen,

I have seen your comments in one of the case similar to this that "We have confirmed aforementioned solution to be appropriate and sufficiently backwards compatible and just released this fix as part of Tasks for AWS 2.4.3"

https://utoolity.atlassian.net/browse/UAA-29.

I have checked we are using 2.14.0 version of Identity Federation for AWS and that shows expired. Is it the reason this app is not working to fetch credentials? If yes then why we get the credentials intermittently.  Also see Task for AWS is having 2.21.1 version. So do we need to update them ?

One more thing to ask where to set these variable in bamboo. I tried to set them as Global variable and given a build but that shows default maxErrorRetry as 7.

bamboo.custom.aws.maxErrorRetry
bamboo.custom.aws.awaitTransitionInterval

Steffen Opel [Utoolity] · Answer

Hi Akshay,

Welcome to the Atlassian Community!

We have not encountered this particular error either yet, and there are only two related issues/discussions for the AWS SDK for Java that we use for the AWS integration, though one comment provides a clue and potential workaround:

I added retries with backoff delay and this solves my problem.

This seems to suggest that "IAM throttling due to huge API calls like 10 per sec" might indeed be the culprit here. Our apps already override the default AWS SDK exponential backoff configuration to better match the CI/CD use case, but maybe your build patterns have intensified and are more likely to trigger a throttling race condition by the AWS APIs?

Potential workaround

Either way, you can try to address this by overriding the resp. Bamboo variables (either globally, per project, or just for the offending plan), as detailed in Configuring the AWS Client:

The increased defaults should be sufficient for most scenarios, but can be adjusted by defining and thus overriding one or both of the following Bamboo variables, either globally, or for a specific project, plan, or custom build:

* bamboo.custom.aws.maxErrorRetry custom.aws.maxErrorRetry – how many retries should the exponential backoff algorithm perform (default: 7)

* custom.aws.awaitTransitionInterval custom.aws.awaitTransitionInterval – how long should the task wait before querying the resource transition state again (default: 15000 milliseconds)

Please let us know how it goes - if this does not address the issue, it would be great if you could sign up on our support site so that we can collaborate on diagnosing this potentially complex intermittent issue.

Cheers,
Steffen