Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Writing files permission in bitbucket-pipeline

Hello,

I'm using bitbucket-pipeline for system tests.

One of the test script (python) is calling:  os.system("python3 run_system.py')

run_system.py - is a script that walks through files in the input folder, then writes file into the output folder.

When I run the system test locally, on my computer, it run without any problem and the test passes.

When I run the system test on bitbucket-pipelines - it fails to write the file into the output folder. What made me think this is a permissions issue.

* Note that if I run run_system.py directly on bitbucket-pipelines (running the script itself and not the test script that runs run_system.py using os.system) - it runs perfectly, and writes file without a problem.
The problem occurs only when I use os.system.

* I tried using subprocess.Popen instead of os.system - same problem

1 answer

0 votes
Caroline R Atlassian Team Dec 07, 2021

Hi, @Dorin Zohar! Welcome to the community! 

In order to further investigate this issue, could you please confirm:

  • What is the exact error you are getting when running the test script in Pipeline?

  • How are you running the test script in the Pipeline? Could you please share the syntax of how you are running it?

  • Will it be possible to share the Pipeline config, test script, and system script with us? Please remove the sensitive data from the scripts before sharing it with us.

Looking forward to hearing from you. 

Kind regards,
Caroline

@Caroline R  thanks, as for your questions:

  • I don't get a specific error - it is just a test and it fails. When I tried to investigate why did it fail, I found out the output files weren't written.
  • The test is running using pylint (see config file)
  • Config is attached
  • Tests are also attached, for simplicity I added only the relevant test (test_system_clean.py) and the runner script (clean_runner.py) with example configuration (clean_config.yaml). 
    It depends on other components (not attached) but the flow is:
    • Creating 10 files in input dir (files created successfully)
    • Creating "test" config (with a structure similar to clean_config.yaml)
    • Running: 
      os.system('python3 ./tools/clean_runner.py')

      from inside the test

    • The clean_runner.py do whatever to the files, then needs to save the new files into the output dir (configured in the config.yaml)

bitbucket-pipelines.yml

definitions:
steps:
- step: &run-test # step id
script: # shell commands to run
- sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
- pip install -r requirements_without_infra.txt
- curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
- apt-get install git-lfs
- pip install pytest-cov
# run the tests and report the code coverage
- python -m pytest --cov-report term-missing --cov=. --junitxml=./test-reports/junit.xml ./tests ./tests_solutions

- step: &run-pylint-pr
script:
- sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
- pip install -r requirements_without_infra.txt
- pip install pytest-pylint
- curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
- apt-get install git-lfs
- git checkout $BITBUCKET_PR_DESTINATION_BRANCH
- git checkout $BITBUCKET_BRANCH
- diff_files=$(git diff-tree -r --diff-filter=D $BITBUCKET_BRANCH $BITBUCKET_PR_DESTINATION_BRANCH --name-only ":(exclude)setup.py" ":(exclude)autodoc.py" | grep -E \.py$ | tr "\n" " ")
- echo ${diff_files} # print the files that were changed
- python -m pytest --junitxml=./test-reports/junit.xml --pylint ${diff_files}



clone:
depth: full
# you can clone lfs files if you set lfs: true

options:
max-time: 7 # maximum minutes to run the tests

pipelines:
pull-requests: # run when pull request is created (or updated)
"**": # on all branches
- parallel: # steps bellow will be run in parallel
- step:
<<: *run-test # concatenate the setup-test step from the "definitions" section
name: Tests # name to show on the pipelines web page
image: python:3.8.7 # docker to use (from Docker Hub)

clean_config.yml

augmentor:
config:
augmentations:
- flip
- rotation
name: Consecutive
walker:
config:
input_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/input
name: FSWalker
writer:
config:
output_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/output
name: ImageFS

clean_runner.py

import hydra
from omegaconf import DictConfig
from clean_code.manager import Manager


def image_augmentor_runner(config):
manager = Manager(config)
manager.run()


@hydra.main(config_path="configs", config_name="clean_config")
def main(config: DictConfig):
image_augmentor_runner(config)


if __name__ == '__main__':
main()

test_system_clean.py

def test_system(tmp_path):
"""
This is a system test, that checking the output of the entire program (main())
We create input files (known images), and apply the program on them (with the randomness of the augmentations).
Then we check that the output files are as expected:
1. Number of output files == number of input files
2. Size of each output image == size of its corresponding input image
3. Sum of pixels of each output image == sum of pixels of its corresponding input image (our augmentations change
only the order of the pixels, not the values)
"""
input_path = os.path.join(tmp_path, 'input')
os.makedirs(input_path, exist_ok=True)

output_path = os.path.join(tmp_path, 'output')
os.makedirs(output_path, exist_ok=True)

num_files_to_test = 10
test_file_handler = TestFilesHandler(num_files_to_test, input_path)
test_files_dict = test_file_handler.get_path_to_image_dict()

config_handler = TestConfigHandler(tmp_path)
yaml_config = config_handler.test_config

with open('./tools/configs/clean_config.yaml', 'w') as outfile:
yaml.dump(yaml_config, outfile, default_flow_style=False)

os.system('python3 ./tools/clean_runner.py')

assert len(os.listdir(output_path)) == num_files_to_test

for out_image_name in os.listdir(output_path):
out_image_path = os.path.join(output_path, out_image_name)
out_image = cv2.imread(out_image_path)

in_image_path = os.path.join(input_path, out_image_name)
in_image = test_files_dict[in_image_path]

assert np.sum(in_image) == np.sum(out_image)
assert in_image.size == out_image.size
Caroline R Atlassian Team Dec 09, 2021

Hi, @Dorin Zohar

We are not experts in python, but we performed some tests and managed to create files using os.system in Bitbucket, so we believe this is not a permissions issue. In this case, we will share the steps we have followed:

bitbucket-pipelines.yml

definitions:
    steps:
        - step: &run-test  
            script:  
                - python test_system_clean.py
                - cat /tmp/input/test.txt              

pipelines:
    branches:  
        "**":  
                - step:
                    <<: *run-test  
                    name: Tests  
                    image: python:3.8.7  



                    

test_system_clean.py

import os
os.system('python3 clean_runner.py')

clean_runner.py

import os
input_path = os.path.join("/tmp", 'input')
os.makedirs(input_path, exist_ok=True)
f= open(input_path + "/test.txt","w+")
print(input_path + "/test.txt")

for i in range(10):
     f.write("This is line %d\r\n" % (i+1))

I hope this helps, but do let me know if you have questions. 

Also, I see that you have access to workspaces in the Standard plan, so in case this is happening in a repo under these workspaces, you should be able to open a support ticket. 

You can open it via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as the product. Please make sure to inform the corresponding workspace. 

Kind regards,
Caroline

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

📣 Calling Bitbucket Data Center customers to participate in research

Hi everyone, Are you Bitbucket DC customer? If so, we'd love to talk to you! Our team wants to dive deep to understand your long-term plans regarding Bitbucket DC and Atlassian Cloud. Do you plan...

123 views 2 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you