Pipeline hangs when running pytest

Question

I have a rather large test base that takes around 30min to run on my local computer. I'm trying to run this in a pipeline. In principle everything works fine, if I run some tests via pytest they pass and the pipeline ends. If I try to run all tests the pipeline hangs and does not end. This means that there is no more output from the pipeline. The time of the pipeline itself is still counting (updates from time to time), but no new output is shown and the time of the command in the step does not updated anymore. The place of the hang is deterministic and depends on the output of pytest. More output means an earlier hang: * This invocation gets me to around 36% of the tests when the pipeline hangs.pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" --log-level=ERROR The output lists every test, whether it passes or not and logged errors, if any: ...
tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:False] PASSED [ 2%]
tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:True] PASSED [ 3%]
...
* If I change the displayed logging to"no logging" and keep the rest the same, this takes me to 92% before the hang:pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" -p no:logging The output is significantly shorter, since not every test is named:tests/datasrc_tests/test_returns_data.py ...E.. [ 92%]
tests/forecast_tests/test_returnsforecast.py . [ 92%]...
tests/datasrc_tests/test_asset_properties_global.py ..... [ 92%]
tests/datasrc_tests/test_depot_data.py .s... [ 92%]
When the pipeline hangs, the duration of the pytest call is not updated anymore, while the time for the whole pipeline is updated from time to time. If I remove enough tests from the test run, everything works. If I run the removed tests by themselves, it works too. When I run the tests in a local Docker container (https://community.atlassian.com/t5/Bitbucket-questions/How-can-I-debug-my-Bitbucket-Pipelines-build-locally/qaq-p/136594) all tests run an the testsuite ends. Any help would be appreciated. Thanks!

andrey_koltsov · Answer

Cause was in celery tasks calls, I mocked these call, and everything is ok

andrey_koltsov · Answer

Hi I got same issue: ``` Memory usage in megabytes:945USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.1 0.0 2384 764 ? Ss 22:44 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)root 8 0.0 0.0 2384 92 ? S 22:44 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)root 9 0.0 0.0 4048 752 ? S 22:44 0:00 /bin/cat /opt/atlassian/pipelines/agent/tmp/build_resultroot 11 0.1 0.0 2384 764 ? Ss 22:44 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/wrapperScript17673453008567385865.shroot 41 0.0 0.0 2384 764 ? S 22:44 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/buildScript4437735362751544859.shroot 42 0.0 0.0 5484 3312 ? S 22:44 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.shroot 48 0.0 0.0 5484 2392 ? S 22:45 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.shroot 49 0.0 0.0 5484 1708 ? S 22:45 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.shroot 51 8.5 0.4 363640 136696 ? S 22:45 0:10 /usr/local/bin/python /usr/local/bin/pytest -v --junitxml=test-reports/report.xml --log-level=WARNroot 84 0.0 0.0 4044 752 ? S 22:47 0:00 sleep 30root 85 0.0 0.0 9388 3040 ? R 22:47 0:00 ps -aux ```

Theodora Boudale · Answer

Hi Dirk,

I believe that the duration of a command gets updated when there is output, so if there is no further output the number showing the duration of the command won't get updated.

What happens with this build eventually? Does it end with a time out error?

My suggestion would be to include the following commands in your bitbucket-pipelines.yml file, at the beginning of the script where the tests run and then run another build.

- while true; do echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && sleep 30; done &
- while true; do ps -aux && sleep 30; done &

These will show us memory and CPU usage during the step, and might give us an indication on why the build is hanging.

You can then attach the Pipelines log file here so I can take a look (please sanitize any private/sensitive info from the log beforehand).

Alternatively, if you don't want to share the log here, you can also open a support ticket for this issue via https://support.atlassian.com/contact/ and provide also the URL of the build. The support team will be able to access the repo and any Pipelines build logs to investigate this issue, if there is a ticket open.

Kind regards,
Theodora