problem with a long running python unit test

[tomeks86]

I have a problem with a failing python unit test in the bitbucket pipeline. The test is a long running one - almost 1.5 s locally and with assertions checking files on the disk. It passes both locally and in the locally run docker container.

In the logs I found that it first checks the assertions and then the tested method writes the files. Is there any implicit asynchronicity in the bitbucket pipelines?

Thank you in advance for your expertise,

Tomasz

[Philip Hodder · Accepted solution]

Hi Tomasz,

Can you include some more information about the failing unit test? What's the error, and what is it testing? Is it doing any multithreaded operations? (As we've found this is often a source of inconsistencies in results.)

The build script itself runs synchronously.

Thanks,

Phil

[tomeks86]

Hello Phil,

I re-enabled the failing test and you can find the log with the failing test here:
https://bitbucket.org/tomeks86/chi1chi2/addon/pipelines/home#!/results/86

There are no multithreaded operations involved, at least to my knowledge.

I'll be glad if you can find an answer,

Tomasz

[Philip Hodder]

Hmmmm. Tricky...

Try the following config, lets see if the files actually get created after the tests finish.

image: tomeks86/python-openbabel

pipelines:
  default:
    - step:
        caches:
          - pip
        script:
          - pip install -r requirements_test.txt
          - flake8 --ignore E501,E126 chi1chi2/
          - python -m unittest discover tests/
        after-script:
          - ps # Check to see if python processes are still running.
          - ls -R # Lists all files. Check to see if necessary tests files are actually created.

I've added an after-script. Which will always run after you main script completes (regardless of if it succeeds or fails). 

[tomeks86]

I tried it:

https://bitbucket.org/tomeks86/chi1chi2/addon/pipelines/home#!/results/95

as well as listing the /tmp directory in the build step:

https://bitbucket.org/tomeks86/chi1chi2/addon/pipelines/home#!/results/96

Nothing interesting unfortunately...

[Philip Hodder]

Do the tests clean themselves up after completing (which may explain why the directories don't exist in the after-script)? Looks like the files aren't there in either pipelines you linked.

Lets try something else, and watch to see if the files are ever created throughout the tests.

image: tomeks86/python-openbabel

pipelines:
  default:
    - step:
        artifacts:
          - file-logs.txt
        caches:
          - pip
        script:
          - while true; do date && ls -Rl /tmp && echo "" && sleep 30; done >> file-logs.txt &
          - pip install -r requirements_test.txt
          - flake8 --ignore E501,E126 chi1chi2/
          - python -m unittest discover tests/

You'll then be able to download the logs from the 'Artifacts' tab in the Pipeline Result page.

Do you have any examples of successful pipelines? Or has this been consistently failing?

Just to confirm, when you say you tried the tests in Docker locally, it was equivalent to the steps listed here? https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html

[tomeks86]

The tests were set up to clean after finishing the run. I turned off the cleaning of the temp directory to prove that the files are actually created. This is the result of the modified pipeline with /tmp directory listing after the run:

https://bitbucket.org/tomeks86/chi1chi2/addon/pipelines/home#!/results/100

I have double-checked and can confirm that my local docker run of the script is equivalent to the one described in the documentation link you sent me.

My temporary solution is to skip those tests in the bitbucket pipeline. I am thinking about a way to enabling them on my computer with some parameter. Have you came across such an issue before?

[Philip Hodder]

Sorry for the delayed response Tomasz, I was on annual leave.

Thanks for verifying the files are created and it works locally.

Just to check, these files need to exist in order for the tests to pass? Perhaps there is some race condition going on. Can you try introduce a delay between the files being created and the assertions before performed.

If it were a concurrency issue, I would expect the tests to be flakey instead of consistently failing. Are you relying on any specific computer architecture at all? What're you running locally?

Unfortunately, I've haven't seen anything similar to this issue before.

Thanks,

Phil

[tomeks86]

Hello Philip,

I have finally found the root cause of my problems - poor formatting. It seems that the negative sign before zeroes added when formatting was architecture dependent. I had to enforce removal of the minus sign for zeros and it works fine now.

Thank you very much for your effort and sorry for bothering you with a problem that only appeared to be interesting.

Best regards,

Tomasz

[Philip Hodder]

No worries! The solution is almost always trivial once you know what it is. Finding it is the hard part. :)