Question on accessing to GPU in Pipeline Runners

[Simon Lee]

We are trying to incorporate tests into the Pipeline Runners.  The software requires access to the GPU.  We tried briefly with "docker run --gpus all" and it looks like accessing to the device is not allowed.  Is this a hard restricton in Pipeline, or is there a way to get around it.  

[Theodora Boudale · Accepted solution]

Hi @Simon Lee and welcome to the community.

This is a limitation in Pipelines.

Our development team has been exploring the option to allow access to GPUs, as you can see from one of Justin Thomas's replies here:

• https://community.atlassian.com/t5/Bitbucket-Pipelines-articles/Bitbucket-Pipelines-Runners-is-now-in-open-beta/ba-p/1691022

Kind regards,
Theodora

[Kristinepetrosyan]

@Theodora Boudale any updates on this?

[Theodora Boudale]

Hi @Kristinepetrosyan,

I don't have any update on this yet, you can add yourself as a watcher in the following feature request (by selecting the Start watching this issue link) if you'd like to get notified via email on updates:

• https://jira.atlassian.com/browse/BCLOUD-21459

I would also recommend adding your vote (by selecting the Vote for this issue link).

Kind regards,
Theodora

[Luu Quang Thang]

I can use Bitbucket pipeline with GPU access by following the steps in root machine (which running Runner):

- Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkit 1.14.3 documentation

- Set nvidia to docker default runtime (It likes --runtime=docker). Edit /etc/docker/daemon.json

{
  "default-runtime": "nvidia",
   "runtimes": {
      "nvidia": {
          "path": "/usr/bin/nvidia-container-runtime",
          "runtimeArgs": []
      }
  }
}

Restart the docker service by typing in sudo systemctl restart docker

- Enable container accesses GPUs by the environment variable NVIDIA_VISIBLE_DEVICES (Specialized Configurations with Docker — NVIDIA Container Toolkit 1.14.3 documentation)

# build a image has default support GPU and will use it for pipeline
# in Dockerfile
# ubuntu is an example base image

FROM ubuntu 
ENV NVIDIA_VISIBLE_DEVICES=all

- Test

 ` docker run --rm ubuntu:gpu nvidia-smi ` 

- Update Bitbucket pipeline

pipelines: 
  default: 
    - step: 
      image: 
        name: ubuntu:gpu

[Thebester]

Integrating tests into Pipeline Runners is like adding a dash of magic to your workflow, but GPU access hiccups can be a buzzkill. We gave "docker run --gpus all" a whirl, but it seems the device isn't playing nice. Wondering if this is a Pipeline hard stop or a puzzle with a workaround.

Might be worth a visit to https://anycompares.com/ - they often have nifty tricks up their sleeve for overcoming such hurdles. Your GPU might just need a different dance move!

[a_stashevsky]

I would also like to advocate for GPU access in bitbucket CI/CD processes. @Theodora Boudale 

[Theodora Boudale]

Hi @a_stashevsky,

Thank you for your comment. Please make sure to add your vote in the following feature request (by selecting the Vote for this issue link in that ticket), if you haven't done so already:

• https://jira.atlassian.com/browse/BCLOUD-21459

You are also more than welcome to leave any feedback there. https://jira.atlassian.com/ is our public issue tracker, and it's where our product managers monitor the demand for new features.

Kind regards,
Theodora