Writing files permission in bitbucket-pipeline

[Dorin Zohar]

Hello,

I'm using bitbucket-pipeline for system tests.

One of the test script (python) is calling:  os.system("python3 run_system.py')

run_system.py - is a script that walks through files in the input folder, then writes file into the output folder.

When I run the system test locally, on my computer, it run without any problem and the test passes.

When I run the system test on bitbucket-pipelines - it fails to write the file into the output folder. What made me think this is a permissions issue.

* Note that if I run run_system.py directly on bitbucket-pipelines (running the script itself and not the test script that runs run_system.py using os.system) - it runs perfectly, and writes file without a problem.
The problem occurs only when I use os.system.

* I tried using subprocess.Popen instead of os.system - same problem

[Caroline R]

Hi, @Dorin Zohar! Welcome to the community! 

In order to further investigate this issue, could you please confirm:

• What is the exact error you are getting when running the test script in Pipeline?

• How are you running the test script in the Pipeline? Could you please share the syntax of how you are running it?

• Will it be possible to share the Pipeline config, test script, and system script with us? Please remove the sensitive data from the scripts before sharing it with us.

Looking forward to hearing from you. 

Kind regards,
Caroline

[Dorin Zohar]

@Caroline R  thanks, as for your questions:

• I don't get a specific error - it is just a test and it fails. When I tried to investigate why did it fail, I found out the output files weren't written.
• The test is running using pylint (see config file)
• Config is attached
• Tests are also attached, for simplicity I added only the relevant test (test_system_clean.py) and the runner script (clean_runner.py) with example configuration (clean_config.yaml). 
  It depends on other components (not attached) but the flow is:
  • Creating 10 files in input dir (files created successfully)
  • Creating "test" config (with a structure similar to clean_config.yaml)
  • Running: 

    os.system('python3 ./tools/clean_runner.py')

    from inside the test

  • The clean_runner.py do whatever to the files, then needs to save the new files into the output dir (configured in the config.yaml)

bitbucket-pipelines.yml

definitions:
    steps:
        - step: &run-test  # step id
            script:  # shell commands to run
                - sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
                - pip install -r requirements_without_infra.txt
                - curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
                - apt-get install git-lfs
                - pip install pytest-cov
                # run the tests and report the code coverage
                - python -m pytest --cov-report term-missing --cov=. --junitxml=./test-reports/junit.xml ./tests ./tests_solutions

        - step: &run-pylint-pr
              script:
                  - sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
                  - pip install -r requirements_without_infra.txt
                  - pip install pytest-pylint
                  - curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
                  - apt-get install git-lfs
                  - git checkout $BITBUCKET_PR_DESTINATION_BRANCH
                  - git checkout $BITBUCKET_BRANCH
                  - diff_files=$(git diff-tree -r --diff-filter=D $BITBUCKET_BRANCH $BITBUCKET_PR_DESTINATION_BRANCH --name-only ":(exclude)setup.py" ":(exclude)autodoc.py" | grep -E \.py$ | tr "\n" " ")
                  - echo ${diff_files}  # print the files that were changed
                  - python -m pytest --junitxml=./test-reports/junit.xml --pylint ${diff_files}



clone:
  depth: full
  # you can clone lfs files if you set lfs: true

options:
    max-time: 7 # maximum minutes to run the tests

pipelines:
    pull-requests:  # run when pull request is created (or updated)
        "**":  # on all branches
            - parallel: # steps bellow will be run in parallel
                - step:
                    <<: *run-test  # concatenate the setup-test step from the "definitions" section
                    name: Tests  # name to show on the pipelines web page
                    image: python:3.8.7  # docker to use (from Docker Hub)

clean_config.yml

augmentor:
  config:
    augmentations:
    - flip
    - rotation
  name: Consecutive
walker:
  config:
    input_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/input
  name: FSWalker
writer:
  config:
    output_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/output
  name: ImageFS

clean_runner.py

import hydra
from omegaconf import DictConfig
from clean_code.manager import Manager


def image_augmentor_runner(config):
    manager = Manager(config)
    manager.run()


@hydra.main(config_path="configs", config_name="clean_config")
def main(config: DictConfig):
    image_augmentor_runner(config)


if __name__ == '__main__':
    main()

test_system_clean.py

def test_system(tmp_path):
    """
    This is a system test, that checking the output of the entire program (main())
    We create input files (known images), and apply the program on them (with the randomness of the augmentations).
    Then we check that the output files are as expected:
        1. Number of output files == number of input files
        2. Size of each output image == size of its corresponding input image
        3. Sum of pixels of each output image == sum of pixels of its corresponding input image (our augmentations change
        only the order of the pixels, not the values)
    """
    input_path = os.path.join(tmp_path, 'input')
    os.makedirs(input_path, exist_ok=True)

    output_path = os.path.join(tmp_path, 'output')
    os.makedirs(output_path, exist_ok=True)

    num_files_to_test = 10
    test_file_handler = TestFilesHandler(num_files_to_test, input_path)
    test_files_dict = test_file_handler.get_path_to_image_dict()

    config_handler = TestConfigHandler(tmp_path)
    yaml_config = config_handler.test_config

    with open('./tools/configs/clean_config.yaml', 'w') as outfile:
        yaml.dump(yaml_config, outfile, default_flow_style=False)

    os.system('python3 ./tools/clean_runner.py')

    assert len(os.listdir(output_path)) == num_files_to_test

    for out_image_name in os.listdir(output_path):
        out_image_path = os.path.join(output_path, out_image_name)
        out_image = cv2.imread(out_image_path)

        in_image_path = os.path.join(input_path, out_image_name)
        in_image = test_files_dict[in_image_path]

        assert np.sum(in_image) == np.sum(out_image)
        assert in_image.size == out_image.size

[Caroline R]

Hi, @Dorin Zohar, 

We are not experts in python, but we performed some tests and managed to create files using os.system in Bitbucket, so we believe this is not a permissions issue. In this case, we will share the steps we have followed:

bitbucket-pipelines.yml

definitions:
    steps:
        - step: &run-test  
            script:  
                - python test_system_clean.py
                - cat /tmp/input/test.txt              

pipelines:
    branches:  
        "**":  
                - step:
                    <<: *run-test  
                    name: Tests  
                    image: python:3.8.7  



                    

test_system_clean.py

import os
os.system('python3 clean_runner.py')

clean_runner.py

import os
input_path = os.path.join("/tmp", 'input')
os.makedirs(input_path, exist_ok=True)
f= open(input_path + "/test.txt","w+")
print(input_path + "/test.txt")

for i in range(10):
     f.write("This is line %d\r\n" % (i+1))

I hope this helps, but do let me know if you have questions. 

Also, I see that you have access to workspaces in the Standard plan, so in case this is happening in a repo under these workspaces, you should be able to open a support ticket. 

You can open it via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as the product. Please make sure to inform the corresponding workspace. 

Kind regards,
Caroline