Pipeline fails even if all tests pass

vtvz

I have two equivalent bitbucket  configs. First:

- set +e
# skipped preparation code
- ./vendor/bin/codecept run unit,integration,functional --skip-group images --xml test-reports/report.xml

- CODE=$?
- echo $CODE
- set -e
- exit $CODE

 And second:

# skipped preparation code
- ./vendor/bin/codecept run unit,integration,functional --skip-group images --xml test-reports/report.xml

Looks similar, but in first case I have this output:

+ set +e

+ ./vendor/bin/codecept run unit,integration,functional --skip-group images --xml test-reports/report.xml
Codeception PHP Testing Framework v2.4.2
Powered by PHPUnit 7.1.5 by Sebastian Bergmann and contributors.

# skipped useless data

Time: 4.59 minutes, Memory: 70.00MB

[30;42mOK (154 tests, 580 assertions)[0m
- [1mXML[22m report generated in [33mfile:///opt/atlassian/pipelines/agent/build/tests/_output/test-reports/report.xml[39m

+ CODE=$?

+ echo $CODE
0

+ set -e

+ exit $CODE

And Successful result:

And second case output:

+ ./vendor/bin/codecept run unit,integration,functional --skip-group images --xml test-reports/report.xml
Codeception PHP Testing Framework v2.4.2
Powered by PHPUnit 7.1.5 by Sebastian Bergmann and contributors.

# skipped useless data

Time: 4.03 minutes, Memory: 70.00MB

[30;42mOK (154 tests, 580 assertions)[0m
- [1mXML[22m report generated in [33mfile:///opt/atlassian/pipelines/agent/build/tests/_output/test-reports/report.xml[39

And for unexplained reasons it FAILS!

I don't understand what's going on and why this happens. Is someone can explain me what I do wrong, why pipeline fails even if status code is 0 and all tests pass and how can I fix it? I'll be very grateful for help..

Philip Hodder

I suspect the exit code inconsistency is going to be related to this issue: https://community.atlassian.com/t5/Bitbucket-questions/In-Bitbucket-Pipelines-why-doesn-t-get-the-expected-exit-code/qaq-p/612957?utm_source=atlcomm&utm_medium=email&utm_campaign=immediate_general_answer&utm_content=topic

See if that helps with the exit code issue, then you should be able to debug further.