Does the BitBucket Pipeline Runners work on AWS ECS?

[drichards-alcon]

I am able to get the new BitBucket Pipeline Runners to work by running the docker image locally on my Ubuntu VM, it works great. Now I want to move this to the company's infrastructure on AWS ECS, is this possible?

[Mark C]

Hi @drichards-alcon,

Thanks for reach out to the community.

I can confirm that you can use Pipelines Runners on AWS as long as you're using Linux with x64 architecture.
This is also mentioned in Pipelines runners official documentation.

I also tested this on my end by spinning up an AWS EC2 instance with a Linux environment and Pipelines runners is working great.

Hope it helps and let me know if you have further questions that I can help with.

Regards,
Mark C

[drichards-alcon]

Thanks for the info @Mark C  I have no doubt it would run on an EC2 instance where I install docker, but will it run in an ECS Task container?

My concern is that as it is running docker in docker and needs access to the docker socket and that this would prevent it from running in the ECS model, forcing us to go down the EC2 model. As to add a new ECS Task is easy but to get approval for an EC2 instance is a bit more work.

Thanks 

[Mark C]

Hi @drichards-alcon

To be honest, I haven't come across a Pipelines runners use case with AWS ECS.

However, I've asked this internally.
I'm afraid there could be some constraints running Pipelines runners on AWS ECS specifically with running docker-in-docker service container that requires privileged mode.

Also, I'm not really sure if AWS ECS is allowing this already.

For this, as long as AWS ECS is allowing a docker-in-docker service container with privileged mode, you can use Pipelines runner there.

Regards,
Mark C

[drichards-alcon]

Makes sense, yeah the use case is that we need to access an on-prem credential vault (Saviynt) so not only do we need someplace to run a docker container, it needs to have access to the on-prem servers, which leaves us on AWS :/ 

Sounds like I will move down the EC2 with Docker path, thanks for the update!

[Einar Coutin]

In this AWS blog post it states clearly ECS doesn't support docker in docker for security reasons:

Building Container Images on ECS 

Do not be mislead by the title as the solution provided is separate from the Bitbucket agent, which does require docker in docker. More details in the quote below, but you can read through the introduction for more context.

There are inherent risks involved in both of these approaches. Containers that have access to the host’s Docker daemon or run in privileged mode can also perform other malicious actions on the host. For example, a container with access to the host’s Docker Engine through a mounted Unix socket would have full access to the underlying Docker API. This would give the Container the privileges to start and stop any other container running on that Docker Engine, or even docker exec into other containers. Even in single-tenant ECS clusters, this can lead to severe ramifications as it exposes a back door for hostile actors.

AWS Fargate runs each container in a VM-isolated environment. It also imposes security best practices, including prohibiting running containers from mounting directories or sockets from the underlying host and preventing containers from running with additional linux capabilities or using the --privileged flag. As a result, customers cannot build container images inside Fargate containers using the builder within Docker Engine.

[Roberto Andrade]

Well, it's weird because you can specify that you want to run privileged containers on ECS according to their "container definition" documentation: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html#ECS-Type-ContainerDefinition-privileged

[Einar Coutin]

@Roberto Andrade Nice catch!!

That must be relatively new.

We're still missing the other side of the equation though. There's no REST API to talk to the bitbucket cloud to register runners.

I am aware there's a ticket request for it somewhere and if I recall @Mark C  was also dealing with that request at some point. LMC if I can find that forum thread.

Yes, here it is:
Is there a way to programatically create a new run... (atlassian.com)

You can find the links to the Atlassian team's Jira tickets for that.

[Einar Coutin]

@Hana Kučerová Hello again Hana,

I copy you on this thread because I see you're a COMMUNITY LEAD from your other reply on this thread, thanks for that by the way:
Solved: Build Minutes - are they counted monthly or yearly (atlassian.com)

As for this we have gotten no traction in a while so I'm asking for some help.

Is there any way to know if there is any traction on Bitbucket's team efforts to build a self hosted runner self-service api?

See the link that I posted above:

https://community.atlassian.com/t5/Bitbucket-questions/Is-there-a-way-to-programatically-create-a-new-runner-Not-using/qaq-p/1967083


Right now we have to create the runners manually through the UI only and there's no way to AUTO-SCALE those runners. I copy you on this thread because is ECS is but one way to do that, but the Jira tickets are duplicated:

[BCLOUD-21309] Have public API endpoints for pipelines runners - Create and track feature requests for Atlassian products.

[BCLOUD-21708] API for creating a new Runner - Create and track feature requests for Atlassian products.

But nobody is taking a look at those.

Thank You so Much in advance!!

[Hana Kučerová]

Hi @Einar Coutin ,

please be aware I’m not part of the Atlassian team and they don’t share these kind of information with me.

Being Community leader simply means I’m active member of this forum, who helps other members with their problems a lot.