gcloud app deployment sometimes unresponsive

Matt Haff

I am using Bitbucket Pipelines to automatically deploy updates to my staging environment. Most of the time it completes in < 1 minute, however sometimes it hangs and never completes. I am limiting it to 10 minutes through Bitbucket to avoid the build hanging for longer (in the past it has run for 90+ minutes).

Attached is the output of my Bitbucket Pipeline, there aren't any errors shown besides the timing out. How can I troubleshoot and fix this problem? Right now the only way around it is for me to rerun the failed step until it passes, which sometimes takes 3-4 tries.

App.yaml

runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /(.*\.(html|css|js|png|jpg|woff|json))
  static_files: build/\1
  upload: build/(.*\.(html|css|js|png|jpg|woff|json))
- url: /.*
  static_files: build/index.html
  upload: build/index.html
- url: /
  static_dir: build
skip_files:
- src/
- coverage/
- documentation/
- node_modules/
- __mocks__/
- ^\.editorconfig/.*
- ^\.git/.*
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
- ^(.*/)?.*\.bak$

bitbucket-pipelines.yml

image: node:9.11.2

options:
  max-time: 10
pipelines:
  default:
    - step:
        name: Build Application
        caches:
          - node
        script:
          - npm install
          - npm run build
        artifacts:
          - build/**
    - step:
        name: Deploy to Staging
        deployment: staging
        script:
          # Set a couple variables for readability
          - SDK_VERSION=197.0.0
          - SDK_FILENAME=google-cloud-sdk-${SDK_VERSION}-linux-x86_64.tar.gz
          # Install Google Cloud SDK
          - curl -o /tmp/google-cloud-sdk.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/${SDK_FILENAME}
          - tar -xvf /tmp/google-cloud-sdk.tar.gz -C /tmp/
          - /tmp/google-cloud-sdk/install.sh -q
          - source /tmp/google-cloud-sdk/path.bash.inc
          - gcloud -v
          # Authenticating with the service account key file
          - echo $GCLOUD_API_KEYFILE | base64 --decode --ignore-garbage > ./gcloud-api-key.json
          - gcloud auth activate-service-account --key-file gcloud-api-key.json
          # Linking to the Google Cloud project
          - gcloud config set project $GCLOUD_PROJECT
          # Deploying the application
          - gcloud app deploy --version staging

 

Philip Hodder

Hi Matt,

One thing you might want to try is monitoring the process information, to see if any processes are hanging or becoming zombies.

The relevant documentation is here: https://confluence.atlassian.com/bitbucket/how-to-generate-support-logs-in-your-pipeline-959288939.html

For your bitbucket-pipeline.yml, it would look something like this.

image: node:9.11.2

options:
  max-time: 10
pipelines:
  default:
    - step:
        name: Build Application
        caches:
          - node
        script:
          - npm install
          - npm run build
        artifacts:
          - build/**
    - step:
        name: Deploy to Staging
        deployment: staging
        artifacts:
          - process-logs.txt # Process information from the build
        script:
          # Dump process information into a file
          - while true; do date && ps aux && echo "" && sleep 30; done >> process-logs.txt &
          # Set a couple variables for readability
          - SDK_VERSION=197.0.0
          - SDK_FILENAME=google-cloud-sdk-${SDK_VERSION}-linux-x86_64.tar.gz
          # Install Google Cloud SDK
          - curl -o /tmp/google-cloud-sdk.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/${SDK_FILENAME}
          - tar -xvf /tmp/google-cloud-sdk.tar.gz -C /tmp/
          - /tmp/google-cloud-sdk/install.sh -q
          - source /tmp/google-cloud-sdk/path.bash.inc
          - gcloud -v
          # Authenticating with the service account key file
          - echo $GCLOUD_API_KEYFILE | base64 --decode --ignore-garbage > ./gcloud-api-key.json
          - gcloud auth activate-service-account --key-file gcloud-api-key.json
          # Linking to the Google Cloud project
          - gcloud config set project $GCLOUD_PROJECT
          # Deploying the application
          - gcloud app deploy --version staging

You can then download the artifact and view the process information over time. That may reveal some more information for you to debug with.

Thanks,

Phil