How to generate support logs in your pipelines

[#1]

If you hit an issue using pipelines, the logs can give you great information on what might be happening. Rather than try and read through all the logs created you can use the power of artifacts to create specific logs, which you can then download. 

The benefit of this is that it keeps your main build log relevant to the build information. You only need to download the extra support logs when you need in depth information into your pipelines environment.

For example you can get a snapshot of what processes are running in your pipeline by adding the lines below to your bitbucket-pipelines.yml. This is useful if you’re trying to see if a process has hung or is using too much memory.

pipelines:
  default:
    - step:
        artifacts:
          - process-logs.txt # Declaring that you want to keep this as an artifact
        script:
          - while true; do date && ps aux && echo "" && sleep 30; done >> process-logs.txt &
          - # The rest of the script.

This will log the process information to the process-logs.txt file, instead of the build logs.

Once the pipeline step completes, you can download the artifact from the Artifacts tab on the Result Page.

Other examples

Container memory monitoring

This is useful if you are getting out of memory errors on your build container, or to check on how much memory your pipeline uses:

pipelines:
  default:
    - step:
        artifacts:
          - memory-logs.txt
        script:
          - while true; do date && echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && echo "" && sleep 30; done >> memory-logs.txt &
          - # The rest of your script.

Docker events

If you’d like to collect more information about your Docker commands, you can log the output of the docker events command:

pipelines:
  default:
    - step:
        services:
          - docker
        artifacts:
          - docker-event-logs.txt
        script:
          - docker events >> docker-event-logs.txt &
          - # The rest of your script.

Combined logs

You can set up a selection of logs at the same time. This example shows 3 logs being extracted.

pipelines:
  default:
    - step:
        services:
          - docker
        artifacts:
          - process-logs.txt
          - memory-logs.txt
          - docker-event-logs.txt
        script:
          - while true; do date && ps -aux && sleep 30 && echo ""; done >> process-logs.txt &
          - while true; do date && echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && echo "" && sleep 30; done >> memory-logs.txt &
          - docker events >> docker-event-logs.txt &
          - # The rest of your script.

Notes

While secured variables output in the UI logs are always masked, if you need to investigate any environment variables, do not redirect secured variables to a file. Redirecting secured environment variables to a file will reveal them. 

We've included all of the contents of this article in our documentation.

For more information about artifacts, check out our  documentation on artifacts.

Build your own

These examples are just the beginning. You can combine a variety of commands to get detailed logs for your situation. We recommend that for ease of viewing and to avoid artifact size limits, that you keep the size of log files below 1GB.

The Bitbucket Pipelines team would love to hear about what ideas you may have for other support scripts! Perhaps you know how to improve the scripts I've shared. Or if you've set anything similar up, feel free to comment here with what you've configured.