How to chown or write to my pipeline container for testing?

[alanhardin]

I have a docker image I am running to test our web application. The problem comes from a write permission 

docker-compose run --rm -v="$BITBUCKET_CLONE_DIR:/var/www/app" -u root -p 0.0.0.0:8000:80 -p 0.0.0.0:8080:8080 -p 0.0.0.0:8081:8081 app ci/run-behat-tests.sh

I do not get an error with attaching the volume. In the script I trying and run a test and I get a write permission error 

<b>Fatal error</b>: Uncaught RuntimeException: Unable to create the storage directory (/var/www/app/var/cache/test/profiler).

In the script I execute the following and get the following results:

echo $(whoami)

root

echo $(pwd)

/var/www/app

echo $(ls -l /var/www/app)

-rw-rw-rw-. 1 nobody nogroup 101 Apr 11 16:58 file
-rw-rw-rw-. 1 nobody nogroup 2627 Apr 11 16:58 file 
-rw-rw-rw-. 1 nobody nogroup 36300 Apr 11 16:58 file 
-rw-rw-rw-. 1 nobody nogroup 4163 Apr 11 16:58 file 
-rw-rw-rw-. 1 nobody nogroup 149 Apr 11 16:58 file 
-rw-rw-rw-. 1 nobody nogroup 105 Apr 11 16:58 file 
-rw-rw-rw-. 1 nobody nogroup 8578 Apr 11 16:58 file 
drwxrwxrwx. 5 nobody nogroup 4096 Apr 11 16:58 app

echo $(ls -l /var/www/app/var)

-rw-rw-rw-. 1 nobody nogroup 33831 Apr 11 17:01 file 
-rw-r--r--. 1 root root 42834 Apr 11 17:01 file
drwxr-xr-x. 3 root root 4096 Apr 11 17:01 cache
drwxrwxrwx. 2 nobody nogroup 4096 Apr 11 16:58 logs
drwxrwxrwx. 2 nobody nogroup 4096 Apr 11 16:58 sessions

How can my application write to my var/cache folder?

[Christian Sciberras]

I've been fighting with this for a while and I couldn't find a solution.

It seems that bitbucket-created directories, (in my case, to be used for caching), are somehow limited from being shared within docker. Changing the user/perm of the directory from outside and inside docker didn't work.

This is especially stupid considering that the caching functionality and docker should go hand-in-hand.

[Philip Hodder]

Hello Alan,

Due to us having Docker user namespaces enabled, you will need to have fairly loose permissions when writing to Docker containers.

In this case, the "/var/www/app/var/caches" directory is only writable by the "owner" of the directory. If you set the permissions to allow "other" (i.e. anyone) to be able to write to that directory, then you should be able to run your build as expected.

Thanks,

Phil

[alanhardin]

Philip,

Thank you for your reply! Should I make the directory writable before I do the

- docker-compose run 

step or in the 

ci/run-behat-tests.sh

script.

[Philip Hodder]

Before the Docker-Compose. I suspect you'll need to make modifications to the Docker images that are being created, before they are referenced in your build. If you're building your own, you'll just need to add an instructions like:

RUN chmod 777 /var/www/app/var/cache

To the Dockerfile.

If you're using a public image, can you share what the image is, and we can look at alternative strategies.

[alanhardin]

I added the chmod right after the image was built:

- chmod 777 /var/www/app/var/cache

And I got the following error:

chmod: cannot access '/var/www/app/var/cache': No such file or directory

Next I tried using the Bitbucket_Clone_Dir variable in the directory path

- chmod 777 $BITBUCKET_CLONE_DIR/var/cache

And got the following error:

chmod: cannot access '/opt/atlassian/pipelines/agent/build/var/cache': No such file or directory

Then I added the chmod inside the script that I am doing the docker-compose run on before I do my tests and get the following some error as before:

<b>Fatal error</b>: Uncaught RuntimeException: Unable to create the storage directory (/var/www/app/var/cache/test/profiler).

Lastly I went up a directory and tried to chmod recursively:

chmod -R 777 /var/www/app/var

and got the following error:

chmod: changing permissions of '/var/www/app/var': Operation not permitted 

I am using public images they are:

The pipeline image is 

php:7.3-apache-stretch

The step image is

atlassian/default-image:2

@Philip Hodder Please help you are my only hope!

[Philip Hodder]

Hey Alan,

Sorry for the delay getting back to you.

Yeah, looks like the chmod commands wont work in this case. So lets not try that anymore. :)

Can I get a little bit more information to debug with?

Can you paste in an example of your docker-compose.yml? (Feel free to replace any sensitive data with dummy values)

Can you also paste in an example of your ci/run-behat-tests.sh file?

Would you also be able to run any commands in your script with verbose mode? The RuntimeException is lacking some information that would be helpful. In particular, I'd like to know if that is due to permission errors, or something else.

Thanks,

Phil

[alanhardin]

 bitbucket-pipeline.yaml

image: php:7.3-apache-stretch
clone:
depth: full
pipelines:
branches:
'{master,pipeline-test}':
- step:
name: Install Dependencies and Test
image: atlassian/default-image:2
caches:
- pip
- node
- docker
script:
- curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
- python get-pip.py
- pip install docker-compose
- chmod -R +x ci
- sh ci/env-setup.sh
- docker-compose build
- chown -R www-data .
- echo $BITBUCKET_CLONE_DIR
- ls -al $BITBUCKET_CLONE_DIR
- ls -al
- docker-compose run --rm -v="$BITBUCKET_CLONE_DIR:/var/www/app" -u root -p 0.0.0.0:8000:80 -p 0.0.0.0:8080:8080 -p 0.0.0.0:8081:8081 app ci/run-behat-tests.sh
services:
- docker

docker-compose.yaml

version: "2.1"

services:
selenium:
image: selenium/standalone-chrome
volumes:
- /dev/shm:/dev/shm

app:
build: .
ports:
- "0.0.0.0:8000:80"
- "0.0.0.0:8080:8080"
- "8081:8081"
volumes:
- .:/var/www/app:cached
environment:
STUFF: abc
links:
- mysql
- rabbitmq
- redis
- selenium
command: "/pathto/web-startup.sh"

worker:
build: .
volumes:
- .:/var/www/app:cached
environment:
STUFF: abc
links:
- mysql
- rabbitmq
- redis
command: "/pathto/worker-startup.sh"

blackfire:
image: blackfire/blackfire
env_file: .env

frontend:
build: .
volumes:
- .:/var/www/app:cached
command: "/pathto/yarn-startup.sh"

redis:
image: redis

mysql:
image: mysql:5
ports:
- "3306:3306"
environment:
PASSWORD: password
USER: user

rabbitmq:
image: rabbitmq:3.6-management
environment:
STUFF: default
ports:
- 5672:5672
- 15672:15672






Dockerfile

FROM php:7.2.3-apache-stretch

RUN installing various items

COPY docker/file.txt /pathto/file.txt

COPY . /var/www/app
WORKDIR /var/www/app

RUN pathto/build-args.sh

RUN chown -R www-data var/*
RUN chown -R 777 var/*

RUN pwd

run-behat-tests.sh

#!/usr/bin/env bash

set -eu

echo "behat test"

echo "adding ServerName localhost to apache2.conf"

cd /var/www/app

echo "running server in background"

./pathto/serverrun.sh

until $(curl --output /dev/null --silent --head --fail http://localhost:8081); do
printf '.'
sleep 5
done

echo "connection made"

echo "Loading fixtures"

./pathto/runfixtures.sh

chmod -R 777 /var/www/app/var

echo "checking permissions on app folder"
echo $(ls -l /var/www/app)

echo "checking permissions on cache folder"
echo $(ls -l /var/www/app/var)

echo $(pwd)

echo $(whoami)

echo "running behat test"

php /pathto/behat "features/somefeature.feature":10

echo "killing container"

kill -s SIGKILL 1

 Should I use a different Image? would that allow me to change permissions in the var directory? I have also tried the chmod -R 777 var in different files like the bitbucket-pipeline.yaml and run-behat-test.sh. 

The results from this setup is a successful run, but I get an error from my application running behat because it does not have permissions to write to the var directory.

<b>Fatal error</b>: Uncaught RuntimeException: Unable to create the storage directory (/var/www/app/var/cache/test/profiler).

I'm not sure what I need to run in verbose mode or how.

@Philip Hodder thank for your help once again!

[Philip Hodder]

@alanhardin hmmm. Okay, lets work through a few things.

1. Can I get you to add

mkdir -p /var/www/app/var/cache/test/profiler

inside of your run-bahat-tests.sh. Just after you run whoami should be fine.

2. If 1 returns permissions issues, can you then move that command before you run "chmod -R 777 /var/www/app/var". I'd like you to make sure you do this separately to 1.

3. "Unable to create the storage directory" doesn't necessarily mean it's permission issues. It seems likely, but it would be good to get something to verify that for sure. Any ideas where you could get some additional logs for that error message?

4. I think your Docker image should be fine.

5. I also couldn't find a verbose mode setting (or a way to alter the log level) in behat or php. 

[alanhardin]

@Philip Hodder I can't thank you enough for helping me.

1) I added 

mkdir -p /var/www/app/var/cache/test/profiler

After whoami inside of run-behat-tests.sh and I received the following error:

<b>Fatal error</b>: Uncaught InvalidArgumentException: The directory &quot;/var/www/app/var/cache/test/easy_admin&quot; does not exist and could not be created. in /var/www/app/vendor/doctrine/cache/lib/Doctrine/Common/Cache/FileCache.php:79

 2) I move the mkdir command above chmod and I ran the pipeline... then I realized that I copied the chmod

chmod -R 777 /var/www/app/var

in my last reply and I did not mean to do that. I have tried to run the chmod statement and when I do I get the following error:

chmod: changing permissions of '/var/www/app/var': Operation not permitted

chmod: changing permissions of '/var/www/app/var/sessions': Operation not permitted

chmod: changing permissions of '/var/www/app/var/sessions/.gitkeep': Operation not permitted

chmod: changing permissions of '/var/www/app/var/SymfonyRequirements.php': Operation not permitted

chmod: changing permissions of '/var/www/app/var/logs': Operation not permitted

chmod: changing permissions of '/var/www/app/var/logs/.gitkeep': Operation not permitted

3) I tried to use behat --expand but received the following error:

The "--expand" option does not exist.

[Philip Hodder]

No worries. We'll get there! 🕵️‍♂️

Lets try something else. What you got in the output of 1 could actually be a small lead. As the directory that failed to be created was now a different one than before (as that one now exists). It looks like we can create the directories though Bash commands. But behat is doing something differently which causes issues.

Can I get you to add an after-script to dump the final state of your directory. please keep the mkdir command from (1) in.

after-script:
  - ls -Rla $BITBUCKET_CLONE_DIR

Also, just to try something that I think shouldn't matter, but just in case. Can I get you to explicitly mount your Docker volume as read-write:

docker-compose run --rm -v="$BITBUCKET_CLONE_DIR:/var/www/app:rw" -u root -p 0.0.0.0:8000:80 -p 0.0.0.0:8080:8080 -p 0.0.0.0:8081:8081 app ci/run-behat-tests.sh

This should already be defaulting to being writable. But juuuuuuust in case, lets check. :)

[alanhardin]

@Philip Hodder sorry for the delay. I got the same error when adding the read write (:rw) to the volume.

<b>Fatal error</b>: Uncaught InvalidArgumentException: The directory &quot;/var/www/app/var/cache/test/easy_admin&quot; does not exist and could not be created. in /var/www/app/vendor/doctrine/cache/lib/Doctrine/Common/Cache/FileCache.php:79

The after-script output of the for /var/www/app/var folder is:

/opt/atlassian/pipelines/agent/build/var:
total 100
drwxrwxrwx. 5 www-data root 4096 May 25 19:54 .
drwxrwxrwx. 20 www-data root 4096 May 25 19:54 ..
-rw-rw-rw-. 1 www-data root 33831 May 25 19:54 SymfonyRequirements.php
-rw-r--r--. 1 165536 165536 42834 May 25 19:54 bootstrap.php.cache
drwxr-xr-x. 4 165536 165536 4096 May 25 19:54 cache
drwxrwxrwx. 2 www-data root 4096 May 25 19:49 logs
drwxrwxrwx. 2 www-data root 4096 May 25 19:49 sessions

/opt/atlassian/pipelines/agent/build/var/cache:
total 16
drwxr-xr-x. 4 165536 165536 4096 May 25 19:54 .
drwxrwxrwx. 5 www-data root 4096 May 25 19:54 ..
drwxr-xr-x. 8 165536 165536 4096 May 25 19:54 dev
drwxr-xr-x. 8 165536 165536 4096 May 25 19:54 test

/opt/atlassian/pipelines/agent/build/var/cache/test:
total 2520
drwxr-xr-x. 8 165536 165536 4096 May 25 19:54 .
drwxr-xr-x. 4 165536 165536 4096 May 25 19:54 ..
drwxr-xr-x. 2 165536 165536 69632 May 25 19:54 ContainerOgytmox
-rw-r--r--. 1 165536 165536 7743 May 25 19:54 annotations.map
-rw-r--r--. 1 165536 165536 778 May 25 19:54 appTestDebugProjectContainer.php
-rw-r--r--. 1 165536 165536 236422 May 25 19:54 appTestDebugProjectContainer.php.meta
-rw-r--r--. 1 165536 165536 1226378 May 25 19:54 appTestDebugProjectContainer.xml
-rw-r--r--. 1 165536 165536 236422 May 25 19:54 appTestDebugProjectContainer.xml.meta
-rw-r--r--. 1 165536 165536 721387 May 25 19:54 appTestDebugProjectContainerCompiler.log
-rw-r--r--. 1 165536 165536 39337 May 25 19:54 appTestDebugProjectContainerDeprecations.log
drwxr-xr-x. 3 165536 165536 4096 May 25 19:54 doctrine
drwxr-xr-x. 3 165536 165536 4096 May 25 19:54 fervoenumbundle
drwxr-xr-x. 2 165536 165536 4096 May 25 19:54 jms_serializer
drwxr-xr-x. 5 165536 165536 4096 May 25 19:54 pools
drwxr-xr-x. 2 165536 165536 4096 May 25 19:54 profiler

please let me know if there is another folder that you want to see from the after script. I have every folder printed out.

165536 seems to be an administrator owner and group... 

In my Docker file I have the following at the end of the file:

RUN chown -R www-data var/*
RUN chmod -R 777 var/*

I'm sure that is php cannot create /var/www/app/var/cache/test/easy_admin

I'm don't know when and why the owner and group were changed to 165536.

[alanhardin]

@Philip Hodder I also tried to use run-as-user after each of my images in the pipeline

image: php:7.3-apache-stretch
run-as-user: 1000

atlassian/default-image:2
run-as-user: 1000

and I get the same out come.

[Philip Hodder]

Hey Alan,

Sorry for the delay, I was on annual leave for the last two weeks.

Have you gotten any more information since your last message?

Looking at what you have so far:

• This definitely appears to be hitting the User Namespace security feature I mentioned earlier. Which is why this issue won't be occurring locally on your machine.
• The issue appears to be that one of your containers is creating a directory, that isn't writable by the other containers. As they are running as different users, and the directories can only be written by the owner user. Then another container/user attempts to write to it, which then causes the error. When we created the directory outside of the Docker compose, it worked as we also made that directory writable to all users.
• We may be able to work around this by manipulating the umask inside of the containers. There's a relevant feature request in Docker. But we'll have to work around this in another way for now (as it doesn't seem like it'll get closed any time soon).

Lets just do a quick investigation first. Can I get you to prefix, inside of your docker-compose.yml file, each command with 'umask &&'

i.e.:

• command: "umask && /pathto/web-startup.sh"
  

• command: "umask && /pathto/worker-startup.sh"

• command: "umask && /pathto/yarn-startup.sh"

And any other command fields I may have missed. Can you let me know the output of each of these umask commands?

Hopefully we're getting closer to solving this! 🤞

[alanhardin]

@Philip Hodder Sorry for the delay. I added

umask && 

to all of my commands in my docker-compose.yml file. There wasn't any output from these commands. I don't see them in the build logs. 

[Philip Hodder]

@alanhardin Bugger. It should be showing on the first line of the output.

I manually checked the umask of the images:

• php:7.3-apache-stretch - 0022
• selenium/standalone-chrome - 0022
• blackfire/blackfire - 0022
• mysql:5 - 0022
• rabbitmq:3.6-management - 0022

Can you try changing the `umask` command to `umask 0000`? That'll change the default permissions on new files created and should hopefully allow for the various containers to write to shared files.

[Santiago M.]

How does this story end?

We also ran into permission issues in our pipeline.

Another team member ran CHMOD 777 locally on the file and fixed the issue but as I'm in Windows I will not be able to run it in our next repository.

@Philip Hodder , could you please help?

[jowanp]

Any movement on this ? it kind of rules out using shared volumes in the pipeline