Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Pipeline hangs when running pytest

I have a rather large test base that takes around 30min to run on my local computer.

I'm trying to run this in a pipeline. In principle everything works fine, if I run some tests via pytest they pass and the pipeline ends.

If I try to run all tests the pipeline hangs and does not end. This means that there is no more output from the pipeline. The time of the pipeline itself is still counting (updates from time to time), but no new output is shown and the time of the command in the step does not updated anymore.

The place of the hang is deterministic and depends on the output of pytest. More output means an earlier hang:

  • This invocation gets me to around 36% of the tests when the pipeline hangs.
    pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" --log-level=ERROR

    The output lists every test, whether it passes or not and logged errors, if any:

    ...
    tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:False] PASSED [ 2%]
    tests/test_BadPerformanceFilter.py::test_BadPerformanceFilter_should_remove_assets_with_bad_performance[quant=9-cat=4-max_cat=100-parallel:True] PASSED [ 3%]
    ...

     

  • If I change the displayed logging to"no logging" and keep the rest the same, this takes me to 92% before the hang:
    pytest -rfE -l --tb=short --disable-warnings --junitxml="./test-results/results.xml" -p no:logging
    The output is significantly shorter, since not every test is named:
    ...
    tests/datasrc_tests/test_asset_properties_global.py
    ..... [ 92%]

    tests/datasrc_tests/test_depot_data.py
    .s... [ 92%]
    tests/datasrc_tests/test_returns_data.py ...E.. [ 92%]
    tests/forecast_tests/test_returnsforecast.py . [ 92%]

When the pipeline hangs, the duration of the pytest call is not updated anymore, while the time for the whole pipeline is updated from time to time.

If I remove enough tests from the test run, everything works. If I run the removed tests by themselves, it works too.

When I run the tests in a local Docker container (https://community.atlassian.com/t5/Bitbucket-questions/How-can-I-debug-my-Bitbucket-Pipelines-build-locally/qaq-p/136594) all tests run an the testsuite ends.

Any help would be appreciated.

Thanks!

2 answers

0 votes

Were you able to figure this out? I've got the same problem.

0 votes

Hi Dirk,

I believe that the duration of a command gets updated when there is output, so if there is no further output the number showing the duration of the command won't get updated.

What happens with this build eventually? Does it end with a time out error?

My suggestion would be to include the following commands in your bitbucket-pipelines.yml file, at the beginning of the script where the tests run and then run another build.

- while true; do echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && sleep 30; done &
- while true; do ps -aux && sleep 30; done &

These will show us memory and CPU usage during the step, and might give us an indication on why the build is hanging.

You can then attach the Pipelines log file here so I can take a look (please sanitize any private/sensitive info from the log beforehand).

Alternatively, if you don't want to share the log here, you can also open a support ticket for this issue via https://support.atlassian.com/contact/ and provide also the URL of the build. The support team will be able to access the repo and any Pipelines build logs to investigate this issue, if there is a ticket open.

Kind regards,
Theodora

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
Community showcase
Posted in Bitbucket

Contest: Share your custom Bitbucket Pipe and win

Announced in this blog, this holiday season we’re celebrating all things CI/CD and between now and the end of 2019 we’ll be showcasing content, use cases, feature announcements and more. One featur...

2,126 views 11 6
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you