gcloud app deployment sometimes unresponsive

[Matt Haff]

I am using Bitbucket Pipelines to automatically deploy updates to my staging environment. Most of the time it completes in < 1 minute, however sometimes it hangs and never completes. I am limiting it to 10 minutes through Bitbucket to avoid the build hanging for longer (in the past it has run for 90+ minutes).

Attached is the output of my Bitbucket Pipeline, there aren't any errors shown besides the timing out. How can I troubleshoot and fix this problem? Right now the only way around it is for me to rerun the failed step until it passes, which sometimes takes 3-4 tries.

App.yaml

runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /(.*\.(html|css|js|png|jpg|woff|json))
  static_files: build/\1
  upload: build/(.*\.(html|css|js|png|jpg|woff|json))
- url: /.*
  static_files: build/index.html
  upload: build/index.html
- url: /
  static_dir: build
skip_files:
- src/
- coverage/
- documentation/
- node_modules/
- __mocks__/
- ^\.editorconfig/.*
- ^\.git/.*
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
- ^(.*/)?.*\.bak$

bitbucket-pipelines.yml

image: node:9.11.2

options:
  max-time: 10
pipelines:
  default:
    - step:
        name: Build Application
        caches:
          - node
        script:
          - npm install
          - npm run build
        artifacts:
          - build/**
    - step:
        name: Deploy to Staging
        deployment: staging
        script:
          # Set a couple variables for readability
          - SDK_VERSION=197.0.0
          - SDK_FILENAME=google-cloud-sdk-${SDK_VERSION}-linux-x86_64.tar.gz
          # Install Google Cloud SDK
          - curl -o /tmp/google-cloud-sdk.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/${SDK_FILENAME}
          - tar -xvf /tmp/google-cloud-sdk.tar.gz -C /tmp/
          - /tmp/google-cloud-sdk/install.sh -q
          - source /tmp/google-cloud-sdk/path.bash.inc
          - gcloud -v
          # Authenticating with the service account key file
          - echo $GCLOUD_API_KEYFILE | base64 --decode --ignore-garbage > ./gcloud-api-key.json
          - gcloud auth activate-service-account --key-file gcloud-api-key.json
          # Linking to the Google Cloud project
          - gcloud config set project $GCLOUD_PROJECT
          # Deploying the application
          - gcloud app deploy --version staging

 

[Philip Hodder]

Hi Matt,

One thing you might want to try is monitoring the process information, to see if any processes are hanging or becoming zombies.

The relevant documentation is here: https://confluence.atlassian.com/bitbucket/how-to-generate-support-logs-in-your-pipeline-959288939.html

For your bitbucket-pipeline.yml, it would look something like this.

image: node:9.11.2

options:
  max-time: 10
pipelines:
  default:
    - step:
        name: Build Application
        caches:
          - node
        script:
          - npm install
          - npm run build
        artifacts:
          - build/**
    - step:
        name: Deploy to Staging
        deployment: staging
        artifacts:
          - process-logs.txt # Process information from the build
        script:
          # Dump process information into a file
          - while true; do date && ps aux && echo "" && sleep 30; done >> process-logs.txt &
          # Set a couple variables for readability
          - SDK_VERSION=197.0.0
          - SDK_FILENAME=google-cloud-sdk-${SDK_VERSION}-linux-x86_64.tar.gz
          # Install Google Cloud SDK
          - curl -o /tmp/google-cloud-sdk.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/${SDK_FILENAME}
          - tar -xvf /tmp/google-cloud-sdk.tar.gz -C /tmp/
          - /tmp/google-cloud-sdk/install.sh -q
          - source /tmp/google-cloud-sdk/path.bash.inc
          - gcloud -v
          # Authenticating with the service account key file
          - echo $GCLOUD_API_KEYFILE | base64 --decode --ignore-garbage > ./gcloud-api-key.json
          - gcloud auth activate-service-account --key-file gcloud-api-key.json
          # Linking to the Google Cloud project
          - gcloud config set project $GCLOUD_PROJECT
          # Deploying the application
          - gcloud app deploy --version staging

You can then download the artifact and view the process information over time. That may reveal some more information for you to debug with.

Thanks,

Phil