Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Pipeline hangs when running pytest

I have a rather large test base that takes around 30min to run on my local computer.

I'm trying to run this in a pipeline. In principle everything works fine, if I run some tests via pytest they pass and the pipeline ends.

If I try to run all tests the pipeline hangs and does not end. This means that there is no more output from the pipeline. The time of the pipeline itself is still counting (updates from time to time), but no new output is shown and the time of the command in the step does not updated anymore.

The place of the hang is deterministic and depends on the output of pytest. More output means an earlier hang:

  • This invocation gets me to around 36% of the tests when the pipeline hangs.
    pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" --log-level=ERROR

    The output lists every test, whether it passes or not and logged errors, if any:

    ...
    tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:False] PASSED [ 2%]
    tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:True] PASSED [ 3%]
    ...

     

  • If I change the displayed logging to"no logging" and keep the rest the same, this takes me to 92% before the hang:
    pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" -p no:logging
    The output is significantly shorter, since not every test is named:
    ...
    tests/datasrc_tests/test_asset_properties_global.py
    ..... [ 92%]

    tests/datasrc_tests/test_depot_data.py
    .s... [ 92%]
    tests/datasrc_tests/test_returns_data.py ...E.. [ 92%]
    tests/forecast_tests/test_returnsforecast.py . [ 92%]

When the pipeline hangs, the duration of the pytest call is not updated anymore, while the time for the whole pipeline is updated from time to time.

If I remove enough tests from the test run, everything works. If I run the removed tests by themselves, it works too.

When I run the tests in a local Docker container (https://community.atlassian.com/t5/Bitbucket-questions/How-can-I-debug-my-Bitbucket-Pipelines-build-locally/qaq-p/136594) all tests run an the testsuite ends.

Any help would be appreciated.

Thanks!

2 answers

0 votes

Hi Dirk,

I believe that the duration of a command gets updated when there is output, so if there is no further output the number showing the duration of the command won't get updated.

What happens with this build eventually? Does it end with a time out error?

My suggestion would be to include the following commands in your bitbucket-pipelines.yml file, at the beginning of the script where the tests run and then run another build.

- while true; do echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && sleep 30; done &
- while true; do ps -aux && sleep 30; done &

These will show us memory and CPU usage during the step, and might give us an indication on why the build is hanging.

You can then attach the Pipelines log file here so I can take a look (please sanitize any private/sensitive info from the log beforehand).

Alternatively, if you don't want to share the log here, you can also open a support ticket for this issue via https://support.atlassian.com/contact/ and provide also the URL of the build. The support team will be able to access the repo and any Pipelines build logs to investigate this issue, if there is a ticket open.

Kind regards,
Theodora

Hi I got same issue:

```

Memory usage in megabytes:945USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.1 0.0 2384 764 ? Ss 22:44 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)root 8 0.0 0.0 2384 92 ? S 22:44 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)root 9 0.0 0.0 4048 752 ? S 22:44 0:00 /bin/cat /opt/atlassian/pipelines/agent/tmp/build_resultroot 11 0.1 0.0 2384 764 ? Ss 22:44 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/wrapperScript17673453008567385865.shroot 41 0.0 0.0 2384 764 ? S 22:44 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/buildScript4437735362751544859.shroot 42 0.0 0.0 5484 3312 ? S 22:44 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.shroot 48 0.0 0.0 5484 2392 ? S 22:45 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.shroot 49 0.0 0.0 5484 1708 ? S 22:45 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.shroot 51 8.5 0.4 363640 136696 ? S 22:45 0:10 /usr/local/bin/python /usr/local/bin/pytest -v --junitxml=test-reports/report.xml --log-level=WARNroot 84 0.0 0.0 4044 752 ? S 22:47 0:00 sleep 30root 85 0.0 0.0 9388 3040 ? R 22:47 0:00 ps -aux
```

Cause was in celery tasks calls, I mocked these call, and everything is ok

Like Theodora Boudale likes this

Were you able to figure this out? I've got the same problem.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

3,741 views 10 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you