Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,298,269
Community Members
 
Community Events
165
Community Groups

Failed to retrieve temporary AWS credentials: com.amazonaws.SdkClientException

Since last few days we have been getting below errors in Bamboo build jobs while performing AWS operations for which we are using Identity Federation for AWS - Connector to fetch the temporary credentials for specific IAM user .

Somehow, It works if we give build after 5-10 minutes.

It was working fine since last week. We have checked that IAM users access key status is Active. Got to know from AWS side that it could be because of IAM throttling due to huge API calls like 10 per sec but still it is not confirmed root cause.

"Failed to retrieve temporary AWS credentials: com.amazonaws.SdkClientException: Unable to execute HTTP request: Remote host closed connection during handshake"

1 comment

Hi Akshay,

Welcome to the Atlassian Community!

We have not encountered this particular error either yet, and there are only two related issues/discussions for the AWS SDK for Java that we use for the AWS integration, though one comment provides a clue and potential workaround:

I added retries with backoff delay and this solves my problem.

This seems to suggest that "IAM throttling due to huge API calls like 10 per sec" might indeed be the culprit here. Our apps already override the default AWS SDK exponential backoff configuration to better match the CI/CD use case, but maybe your build patterns have intensified and are more likely to trigger a throttling race condition by the AWS APIs?

Potential workaround

Either way, you can try to address this by overriding the resp. Bamboo variables (either globally, per project, or just for the offending plan), as detailed in Configuring the AWS Client:

The increased defaults should be sufficient for most scenarios, but can be adjusted by defining and thus overriding one or both of the following Bamboo variables, either globally, or for a specific project, plan, or custom build:

  • bamboo.custom.aws.maxErrorRetry custom.aws.maxErrorRetry – how many retries should the exponential backoff algorithm perform (default: 7)

  • custom.aws.awaitTransitionInterval custom.aws.awaitTransitionInterval – how long should the task wait before querying the resource transition state again (default: 15000 milliseconds)

Please let us know how it goes - if this does not address the issue, it would be great if you could sign up on our support site so that we can collaborate on diagnosing this potentially complex intermittent issue.

Cheers,
Steffen

Like # people like this

Hi Steffen,

I have seen your comments in one of the case similar to this that "We have confirmed aforementioned solution to be appropriate and sufficiently backwards compatible and just released this fix as part of Tasks for AWS 2.4.3"

https://utoolity.atlassian.net/browse/UAA-29.

I have checked we are using 2.14.0 version of Identity Federation for AWS and that shows expired. Is it the reason this app is not working to fetch credentials? If yes then why we get the credentials intermittently.  Also see Task for AWS is having 2.21.1 version. So do we need to update them ?

Screenshot 2021-05-14 at 8.20.53 PM.png

 

Screenshot 2021-05-14 at 8.25.26 PM.png


One more thing to ask where to set these variable in bamboo. I tried to set them as Global variable and given a build but that shows default maxErrorRetry as 7.

bamboo.custom.aws.maxErrorRetry
bamboo.custom.aws.awaitTransitionInterval

Like Steffen Opel _Utoolity_ likes this

Hi Akshay,

Thanks for signing up, I've just made you the reporter of the support request created from your post so we can collaborate on solving this puzzle over there, just quickly:

  • You are using the most current releases of both apps, so no updates are needed right now.
  • An expired license should not matter, because Identity Federation for AWS is free for licensees of other Utoolity apps that integrate them: This works automatically, i.e. Identity Federation for AWS will detect other licensed Utoolity apps and ignore its own license status then – you can simply ignore or remove the expired license, and I'll also provide you a complimentary license with a 100% discount code via the support request to avoid future confusion around this usability issue (unfortunately Atlassian does not support cross-app licenses).
  • The referenced issue UAA-29 likely has an unrelated root cause, see below.

One more thing to ask where to set these variable in bamboo. I tried to set them as Global variable and given a build but that shows default maxErrorRetry as 7.

Unfortunately our documentation erroneously referred to the variables with the bamboo.* prefix required for using variables within the build, which must not be used when defining variables, sorry you had to discover this the hard way! I've updated my answer with the fix to hopefully prevent others from running into this easy to trip over detail.

Fortunately this means the proposed workaround might still solve the intermittent temporary credentials errors, so let's continue the conversation via the support request.

Cheers,
Steffen

Comment

Log in or Sign up to comment
TAGS
Community showcase
Published in Bamboo

Bamboo Data Center on Kubernetes

Hi, If you are running self-managed environments and looking to adopt modern infrastructure, Bamboo Data Center can now be deployed in a Kubernetes cluster. By leveraging Kubernetes, you can easily...

957 views 3 8
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you