I have a rather large test base that takes around 30min to run on my local computer.
I'm trying to run this in a pipeline. In principle everything works fine, if I run some tests via pytest they pass and the pipeline ends.
If I try to run all tests the pipeline hangs and does not end. This means that there is no more output from the pipeline. The time of the pipeline itself is still counting (updates from time to time), but no new output is shown and the time of the command in the step does not updated anymore.
The place of the hang is deterministic and depends on the output of pytest. More output means an earlier hang:
pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" --log-level=ERROR
The output lists every test, whether it passes or not and logged errors, if any:
...
tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:False] PASSED [ 2%]tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:True] PASSED [ 3%]
...
pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" -p no:logging
The output is significantly shorter, since not every test is named:...
tests/datasrc_tests/test_asset_properties_global.py ..... [ 92%]
tests/datasrc_tests/test_depot_data.py .s... [ 92%]tests/datasrc_tests/test_returns_data.py ...E.. [ 92%]
tests/forecast_tests/test_returnsforecast.py . [ 92%]
When the pipeline hangs, the duration of the pytest call is not updated anymore, while the time for the whole pipeline is updated from time to time.
If I remove enough tests from the test run, everything works. If I run the removed tests by themselves, it works too.
When I run the tests in a local Docker container (https://community.atlassian.com/t5/Bitbucket-questions/How-can-I-debug-my-Bitbucket-Pipelines-build-locally/qaq-p/136594) all tests run an the testsuite ends.
Any help would be appreciated.
Thanks!
Hi Dirk,
I believe that the duration of a command gets updated when there is output, so if there is no further output the number showing the duration of the command won't get updated.
What happens with this build eventually? Does it end with a time out error?
My suggestion would be to include the following commands in your bitbucket-pipelines.yml file, at the beginning of the script where the tests run and then run another build.
- while true; do echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && sleep 30; done &
- while true; do ps -aux && sleep 30; done &
These will show us memory and CPU usage during the step, and might give us an indication on why the build is hanging.
You can then attach the Pipelines log file here so I can take a look (please sanitize any private/sensitive info from the log beforehand).
Alternatively, if you don't want to share the log here, you can also open a support ticket for this issue via https://support.atlassian.com/contact/ and provide also the URL of the build. The support team will be able to access the repo and any Pipelines build logs to investigate this issue, if there is a ticket open.
Kind regards,
Theodora
Hi I got same issue:
```
Memory usage in megabytes:
945
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 2384 764 ? Ss 22:44 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)
root 8 0.0 0.0 2384 92 ? S 22:44 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)
root 9 0.0 0.0 4048 752 ? S 22:44 0:00 /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result
root 11 0.1 0.0 2384 764 ? Ss 22:44 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/wrapperScript17673453008567385865.sh
root 41 0.0 0.0 2384 764 ? S 22:44 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/buildScript4437735362751544859.sh
root 42 0.0 0.0 5484 3312 ? S 22:44 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.sh
root 48 0.0 0.0 5484 2392 ? S 22:45 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.sh
root 49 0.0 0.0 5484 1708 ? S 22:45 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript18282977510559340457.sh
root 51 8.5 0.4 363640 136696 ? S 22:45 0:10 /usr/local/bin/python /usr/local/bin/pytest -v --junitxml=test-reports/report.xml --log-level=WARN
root 84 0.0 0.0 4044 752 ? S 22:47 0:00 sleep 30
root 85 0.0 0.0 9388 3040 ? R 22:47 0:00 ps -aux
```
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Cause was in celery tasks calls, I mocked these call, and everything is ok
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Were you able to figure this out? I've got the same problem.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.