Bitbucket pipeline local runner failing randomally

[Brian Schaiper]

I'm appealing to the folks out there in pipeline world.   We have an instance where we have local bitbucket-pipeline-runners in Kubernetes that are meant to be used for building code for a lambda sam deploy.  They are ran via gradle and I'm having some frustrating behavior i'm trying to figure out.   Randomly I get errors. like below.  I can rerun the failed step and then it will rerun and succeed.   At first I thought because we were building things in parallel but when I redid the pipeline file to rune sequentially still received the random failure.   I've had times where it all ran successfully, we've even upgraded gradle to the latest version and still no luck.   While it's great I can manually tell it to rerun the failed step that does not help as far as CICD goes.  Has anyone experienced this and if so how did you fix it / work around it... Also is there a way to have it re-run on failure automatically?   Any help would be appreciated.  If I had hair still i'd be bald.

FAILURE: Build failed with an exception.

87

88

* What went wrong:

89

Execution failed for task ':<app_name>:generateAsc-licenseOpenApiApis'.

90

> A failure occurred while executing io.micronaut.gradle.openapi.tasks.OpenApiClientWorkAction

91

> Could not generate supporting file '/opt/atlassian/pipelines/agent/build/<app_name>/build/generated/openapi/generateAsc-licenseOpenApiApis/.openapi-generator-ignore'

[Theodora Boudale]

Hi Brian,

It is possible that the failure occurs because the process consumes a lot of memory and it reaches the memory limit of the Docker container where the build runs. You can add the following commands in your yml file, in the beginning of the step that intermittently fails:

- while true; do date && ps -aux && sleep 5 && echo ""; done &
- while true; do date && echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && echo "" && sleep 5; done &

These commands will print memory usage in the Pipelines build log for this step, throughout the step. The next time you see a failure after adding these commands, please check in the step's build log if memory usage is close to the 4 GB limit around the time of the failure. Regular steps have 4 GB of limit available, but it is also possible to allocate more:

• https://support.atlassian.com/bitbucket-cloud/docs/step-options/#Size

Other things I would suggest:

• Check the version of the plugin/dependency that is failing, there may be a newer version.
• There may also a bug with the plugin/dependency that is failing. Does it have a public issue tracker where you can check for reported bugs?
• Is it possible to enable debug / verbose output in the command you are running? This will show additional info in the output and perhaps more details on why the file's creation fails.

Kind regards,
Theodora