Spring-Boot native image Build fails with status code 403 forbidden

[Helmut Pasch]

Hi,

I'm trying to build a simple Spring Boot Demo Project as a Spring Native Image and run into this error ...

[INFO] <<< spring-boot-maven-plugin:2.5.6:build-image (default-cli) < package @ demo <<<
[INFO] 
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.5.6:build-image (default-cli) @ demo ---
[INFO] Building image 'docker.io/library/demo:0.0.2-SNAPSHOT'
[INFO] 
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 0%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 5%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 6%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 7%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 9%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 11%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 14%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 16%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 19%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 27%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 46%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 57%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 62%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 70%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 79%
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder:tiny' 100%
[INFO] > Pulled builder image 'paketobuildpacks/builder@sha256:e437b6165f761636304f52cdee66ba6208f8183176711c32005da0a0fe71cce8'
[INFO] > Pulling run image 'docker.io/paketobuildpacks/run:tiny-cnb' 0%
[INFO] > Pulling run image 'docker.io/paketobuildpacks/run:tiny-cnb' 49%
[INFO] > Pulling run image 'docker.io/paketobuildpacks/run:tiny-cnb' 100%
[INFO] > Pulled run image 'paketobuildpacks/run@sha256:033f7b589dde6dbd3c10b25faf5d9ae759a7b4197209f056d43f3168c80c20e9'
[INFO] > Executing lifecycle version v0.14.1
[INFO] > Using build cache volume 'pack-cache-99cdcec8555c.build'
[INFO] 
[INFO] > Running creator
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:51 min
[INFO] Finished at: 2022-08-17T11:21:02Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.5.6:build-image 
(default-cli) on project demo: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.5.6:build-image 
failed: Docker API call to 'localhost:2375/v1.24/containers/create' 
failed with status code 403 "Forbidden" -> [Help 1]

The start seams promising some downloads are running for the buildpacks  ... but then a  failure with 403 forbidden.

My bitbucket-pipelines.yml

image: amazoncorretto:17-alpine-jdk

pipelines:

default:

 - parallel:

 - step:

services:

 - docker

caches:

 - maven

 - docker

name: 'Build and Test'

script:

 - export DOCKER_BUILDKIT=1

 - docker version

 - ./mvnw package spring-boot:build-image

 - step:

name: 'Security scan'

script:

 - echo "Not implemented yet ;-)"

The "docker version" command is only inserted for testing purposes, this commands works well. 

My Spring-Boot Maven POM (snipped) ...

...

<build>

  <plugins>

    <plugin>

       <groupId>org.springframework.boot</groupId>

       <artifactId>spring-boot-maven-plugin</artifactId>

       <configuration>

          <classifier>${repackage.classifier}</classifier>

          <image>

             <builder>paketobuildpacks/builder:tiny</builder>

             <env>

               <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
 
             </env>

          </image>

        </configuration>

    </plugin>

The configuration seems really simple and straightforward from my point of view. 

Does anyone have the same experience or suggestions on what to change to make it work ...

[Theodora Boudale]

Hi Helmut,

Would it be possible to share the logs of the docker daemon from this build? You can find these in the tab named docker in the Pipelines build log.

I believe that the issue is with a custom Auth plugin that we use in Pipelines that simply filters all docker requests sent to the Docker daemon and we disallow any potential malicious/exploitable docker arguments, as described here https://confluence.atlassian.com/bitbucket/run-docker-commands-in-bitbucket-pipelines-879254331.html.

The issue has been reported by other users in our issue tracker here:

https://jira.atlassian.com/browse/BCLOUD-15844

Kind regards,
Theodora

[Helmut Pasch]

Hi Theodora,
yes definitely ...

time="2022-08-17T12:08:38.174595290Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"

time="2022-08-17T12:08:38.174994803Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"

time="2022-08-17T12:08:38.175031169Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"

time="2022-08-17T12:08:39.175172640Z" level=warning msg="Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message" host="tcp://0.0.0.0:2375"

time="2022-08-17T12:08:39.175209733Z" level=warning msg="Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network" host="tcp://0.0.0.0:2375"

time="2022-08-17T12:08:39.175219280Z" level=warning msg="You can override this by explicitly specifying '--tls=false' or '--tlsverify=false'" host="tcp://0.0.0.0:2375"

time="2022-08-17T12:08:39.175226778Z" level=warning msg="Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release" host="tcp://0.0.0.0:2375"

time="2022-08-17T12:08:54Z" level=warning msg="deprecated version : `1`, please switch to version `2`"

time="2022-08-17T12:08:54.438178990Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"

time="2022-08-17T12:08:54.438850571Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"

time="2022-08-17T12:08:54.450405101Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"

time="2022-08-17T12:08:54.450574412Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"

time="2022-08-17T12:08:54.463493722Z" level=warning msg="unable to modify root key limit, number of containers could be limited by this quota: open /proc/sys/kernel/keys/root_maxkeys: no such file or directory"

time="2022-08-17T12:08:54.495324030Z" level=warning msg="Your kernel does not support CPU realtime scheduler"

time="2022-08-17T12:08:54.495375143Z" level=warning msg="Your kernel does not support cgroup blkio weight"

time="2022-08-17T12:08:54.495386027Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"

time="2022-08-17T12:08:54.503315084Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nlsmod: /proc/modules: No such file or directory\nip: can't find device 'br_netfilter'\nlsmod: /proc/modules: No such file or directory\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"

time="2022-08-17T12:08:55Z" level=info msg="Pipelines plugin request authorization." allowed=false method=HEAD plugin=pipelines uri=/_ping

time="2022-08-17T12:08:55.386352930Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "

time="2022-08-17T12:08:55Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping

time="2022-08-17T12:08:55Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/v1.41/version

time="2022-08-17T12:11:22Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=docker.io%2Fpaketobuildpacks%2Fbuilder%3Atiny"

time="2022-08-17T12:11:38Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/builder@sha256:e437b6165f761636304f52cdee66ba6208f8183176711c32005da0a0fe71cce8/json"

time="2022-08-17T12:11:38Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=docker.io%2Fpaketobuildpacks%2Frun%3Atiny-cnb"

time="2022-08-17T12:11:41Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/run@sha256:033f7b589dde6dbd3c10b25faf5d9ae759a7b4197209f056d43f3168c80c20e9/json"

time="2022-08-17T12:11:41Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.24/images/load

time="2022-08-17T12:11:41Z" level=info msg="Container create request." ArgsEscaped=false AttachStderr=false AttachStdin=false AttachStdout=false ExposedPorts="map[]" Healthcheck="<nil>" Labels="map[author:spring-boot]" MacAddress= NetworkDisabled=false OnBuild="[]" OpenStdin=false StdinOnce=false StopSignal= StopTimeout="<nil>" Tty=false plugin=pipelines

time="2022-08-17T12:11:41Z" level=info msg="Container create request." AutoRemove=false BlkioDeviceReadBps="[]" BlkioDeviceReadIOps="[]" BlkioDeviceWriteBps="[]" BlkioDeviceWriteIOps="[]" BlkioWeight=0 BlkioWeightDevice="[]" CPUCount=0 CPUPercent=0 CPUPeriod=0 CPUQuota=0 CPURealtimePeriod=0 CPURealtimeRuntime=0 CPUShares=0 CapAdd="[]" CapDrop="[]" Cgroup= CgroupParent= ConsoleSize="[0 0]" ContainerIDFile= CpusetCpus= CpusetMems= DNS="[]" DNSOptions="[]" DNSSearch="[]" DeviceCgroupRules="[]" Devices="[]" ExtraHosts="[]" GroupAdd="[]" IOMaximumBandwidth=0 IOMaximumIOps=0 Init="<nil>" IpcMode= Isolations= KernelMemory=0 Links="[]" LogConfig="{ map[]}" MaskedPaths="[]" Memory=0 MemoryReservation=0 MemorySwap=0 MemorySwappiness="<nil>" Mounts="[]" NanoCPUs=0 NetworkMode=default OomKillDisable="<nil>" OomScoreAdj=0 PidMode= PidsLimit="<nil>" PortBindings="map[]" Privileged=false PublishAllPorts=false ReadOnlyPaths="[]" RestartPolicy="{ 0}" Runtime= SecurityOpt="[]" ShmSize=0 StorageOpt="map[]" Sysctls="map[]" Ulimits="[]" UsernsMode= VolumeDriver= VolumesFrom="[]" plugin=pipelines

time="2022-08-17T12:11:41Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.24/containers/create

time="2022-08-17T12:11:41.282486139Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"

time="2022-08-17T12:11:41Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-layers-xumbefxjpv?force=1"

time="2022-08-17T12:11:41Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-app-jshifpaglv?force=1"

time="2022-08-17T12:11:41Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/images/pack.local/builder/jfuwjmcokx:latest?force=1"

Maybe the first line is suspicious ...

level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"

[Helmut Pasch]

and this one ...

msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"

[Theodora Boudale]

Hi Helmut,

Thank you for the logs, the last part you posted indicates an issue:

time="2022-08-17T12:11:41.282486139Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"

For the -v option we only support mounting in /opt/atlassian/bitbucketci/agent/build/.* or /opt/atlassian/pipelines/agent/build/.*

Trying to mount volumes outside of these directories is going to give the above error.

I am not very familiar with Spring Boot and I don't know where the docker commands and configuration are saved. If you can find it and change the arguments in -v option to mount inside one of the directories I mentioned, this error shouldn't occur.

Kind regards,
Theodora