Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
Community Members
Community Events
Community Groups

Writing files permission in bitbucket-pipeline


I'm using bitbucket-pipeline for system tests.

One of the test script (python) is calling:  os.system("python3') - is a script that walks through files in the input folder, then writes file into the output folder.

When I run the system test locally, on my computer, it run without any problem and the test passes.

When I run the system test on bitbucket-pipelines - it fails to write the file into the output folder. What made me think this is a permissions issue.

* Note that if I run directly on bitbucket-pipelines (running the script itself and not the test script that runs using os.system) - it runs perfectly, and writes file without a problem.
The problem occurs only when I use os.system.

* I tried using subprocess.Popen instead of os.system - same problem

1 answer

0 votes
Caroline R Atlassian Team Dec 07, 2021

Hi, @Dorin Zohar! Welcome to the community! 

In order to further investigate this issue, could you please confirm:

  • What is the exact error you are getting when running the test script in Pipeline?

  • How are you running the test script in the Pipeline? Could you please share the syntax of how you are running it?

  • Will it be possible to share the Pipeline config, test script, and system script with us? Please remove the sensitive data from the scripts before sharing it with us.

Looking forward to hearing from you. 

Kind regards,

@Caroline R  thanks, as for your questions:

  • I don't get a specific error - it is just a test and it fails. When I tried to investigate why did it fail, I found out the output files weren't written.
  • The test is running using pylint (see config file)
  • Config is attached
  • Tests are also attached, for simplicity I added only the relevant test ( and the runner script ( with example configuration (clean_config.yaml). 
    It depends on other components (not attached) but the flow is:
    • Creating 10 files in input dir (files created successfully)
    • Creating "test" config (with a structure similar to clean_config.yaml)
    • Running: 
      os.system('python3 ./tools/')

      from inside the test

    • The do whatever to the files, then needs to save the new files into the output dir (configured in the config.yaml)


- step: &run-test # step id
script: # shell commands to run
- sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
- pip install -r requirements_without_infra.txt
- curl -s | bash
- apt-get install git-lfs
- pip install pytest-cov
# run the tests and report the code coverage
- python -m pytest --cov-report term-missing --cov=. --junitxml=./test-reports/junit.xml ./tests ./tests_solutions

- step: &run-pylint-pr
- sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
- pip install -r requirements_without_infra.txt
- pip install pytest-pylint
- curl -s | bash
- apt-get install git-lfs
- git checkout $BITBUCKET_BRANCH
- diff_files=$(git diff-tree -r --diff-filter=D $BITBUCKET_BRANCH $BITBUCKET_PR_DESTINATION_BRANCH --name-only ":(exclude)" ":(exclude)" | grep -E \.py$ | tr "\n" " ")
- echo ${diff_files} # print the files that were changed
- python -m pytest --junitxml=./test-reports/junit.xml --pylint ${diff_files}

depth: full
# you can clone lfs files if you set lfs: true

max-time: 7 # maximum minutes to run the tests

pull-requests: # run when pull request is created (or updated)
"**": # on all branches
- parallel: # steps bellow will be run in parallel
- step:
<<: *run-test # concatenate the setup-test step from the "definitions" section
name: Tests # name to show on the pipelines web page
image: python:3.8.7 # docker to use (from Docker Hub)


- flip
- rotation
name: Consecutive
input_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/input
name: FSWalker
output_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/output
name: ImageFS

import hydra
from omegaconf import DictConfig
from clean_code.manager import Manager

def image_augmentor_runner(config):
manager = Manager(config)

@hydra.main(config_path="configs", config_name="clean_config")
def main(config: DictConfig):

if __name__ == '__main__':

def test_system(tmp_path):
This is a system test, that checking the output of the entire program (main())
We create input files (known images), and apply the program on them (with the randomness of the augmentations).
Then we check that the output files are as expected:
1. Number of output files == number of input files
2. Size of each output image == size of its corresponding input image
3. Sum of pixels of each output image == sum of pixels of its corresponding input image (our augmentations change
only the order of the pixels, not the values)
input_path = os.path.join(tmp_path, 'input')
os.makedirs(input_path, exist_ok=True)

output_path = os.path.join(tmp_path, 'output')
os.makedirs(output_path, exist_ok=True)

num_files_to_test = 10
test_file_handler = TestFilesHandler(num_files_to_test, input_path)
test_files_dict = test_file_handler.get_path_to_image_dict()

config_handler = TestConfigHandler(tmp_path)
yaml_config = config_handler.test_config

with open('./tools/configs/clean_config.yaml', 'w') as outfile:
yaml.dump(yaml_config, outfile, default_flow_style=False)

os.system('python3 ./tools/')

assert len(os.listdir(output_path)) == num_files_to_test

for out_image_name in os.listdir(output_path):
out_image_path = os.path.join(output_path, out_image_name)
out_image = cv2.imread(out_image_path)

in_image_path = os.path.join(input_path, out_image_name)
in_image = test_files_dict[in_image_path]

assert np.sum(in_image) == np.sum(out_image)
assert in_image.size == out_image.size
Caroline R Atlassian Team Dec 09, 2021

Hi, @Dorin Zohar

We are not experts in python, but we performed some tests and managed to create files using os.system in Bitbucket, so we believe this is not a permissions issue. In this case, we will share the steps we have followed:


        - step: &run-test  
                - python
                - cat /tmp/input/test.txt              

                - step:
                    <<: *run-test  
                    name: Tests  
                    image: python:3.8.7  


import os

import os
input_path = os.path.join("/tmp", 'input')
os.makedirs(input_path, exist_ok=True)
f= open(input_path + "/test.txt","w+")
print(input_path + "/test.txt")

for i in range(10):
     f.write("This is line %d\r\n" % (i+1))

I hope this helps, but do let me know if you have questions. 

Also, I see that you have access to workspaces in the Standard plan, so in case this is happening in a repo under these workspaces, you should be able to open a support ticket. 

You can open it via, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as the product. Please make sure to inform the corresponding workspace. 

Kind regards,

Suggest an answer

Log in or Sign up to answer

Atlassian Community Events