elastic agents from AMI's containing EBS encrypted with a customer managed kms key?

[James Orange]

Does Atlassian Bamboo version 8.1.1 build 80108 - 14 Dec 21

support deploying Elastic Agents from an AMI with an EBS volume encrypted with a customer managed KMS key? All my attempts result in:

Jun 29, 2023 6:35:00 AM Requested that new elastic instance be created for configuration: **** / ami-****
Jun 29, 2023 6:35:01 AM Elastic instance [i-062f1f71728def7ae] transitioned from STARTING to IDENTIFIED.
Jun 29, 2023 6:35:22 AM Elastic instance [i-062f1f71728def7ae] transitioned from IDENTIFIED to TERMINATED.

I've added permissions to kms:decrypt from the IAM role which the bamboo server assumes. Aslo to the IAM role that the elastic agents themselves assume.

I can successfully deploy elastic agents from an AMI that has an EBS volume encrypted using the default aws managed key.

[alok m]

Based on your description, it seems like you're encountering an issue where Atlassian Bamboo Elastic Agents, specifically when using an AMI with an EBS volume encrypted via a customer-managed AWS KMS key, fail to start properly. Here are some potential causes and troubleshooting steps to help resolve this issue:

1. IAM Permissions:
   • Double-check that the IAM roles associated both with the Bamboo server and the Elastic Agents have the necessary permissions to use the KMS key. This includes not just kms:Decrypt, but also kms:Encrypt, kms:ReEncrypt*, kms:DescribeKey, and kms:GenerateDataKey* as these are often required for operations involving encrypted volumes.
   • Ensure that the IAM roles are properly attached and that there are no policy conflicts or typos.
2. KMS Key Policy:
   • Review the policy attached to your customer-managed KMS key. It needs to explicitly allow the IAM roles used by your Bamboo server and its Elastic Agents to interact with the key. This policy should include permissions for actions such as kms:Encrypt, kms:Decrypt, kms:ReEncrypt*, kms:GenerateDataKey*, and kms:DescribeKey.
   • Make sure the key policy doesn't restrict the key usage to specific resources or conditions that are not being met when Bamboo attempts to launch the Elastic Agents.
3. Networking and Access:
   • Verify that the network configuration (like VPC settings, subnets, security groups) where the Bamboo Elastic Agents are launched allows outbound connections to KMS endpoints. Sometimes, network restrictions can prevent instances from communicating with KMS.
   • Check if there are any VPC endpoint configurations required for accessing KMS securely within your VPC.
4. AMI and EBS Snapshot Permissions:
   • Confirm that the AMI and any associated snapshots are accessible to the account where Bamboo is running. If the AMI or snapshots are shared from another account, ensure that the permissions are set correctly.
   • Verify that the encryption settings of the snapshots used to create the AMI align with what your IAM roles and KMS key policies allow.
5. Logs and Monitoring:
   • Utilize CloudTrail logs to monitor and check for any denied requests or other anomalies that occur when Bamboo attempts to launch the Elastic Agents. This can provide clues as to what might be going wrong.
   • Review the system logs from the instances if they're accessible before termination. Sometimes the instance logs can provide a clue on failures related to filesystem mounting, EBS volume decryption errors, etc.
6. Test with a Simplified Scenario:
   • Try to manually launch an EC2 instance using the same AMI and EBS encryption settings, under the same IAM role as Bamboo uses, to see if the issue replicates. This can help isolate whether the issue is with Bamboo or the AWS configuration.

These steps should help you identify the root cause of the issue and apply the appropriate fixes. If all configurations look correct, it might be a specific issue with how Bamboo handles KMS-encrypted EBS volumes, and Atlassian Support would be the next best resource.