nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Why does serverless-deploy pipe use serverless-python-requirements without poetry installed?

Sam Mahr

I have a serverless app that uses Poetry for package management.

I am using bitbucketpipelines/serverless-deploy:1.5.0 to deploy my serverless app. When I try to deploy, I get this error:

Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
× Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)

Why is serverless-python-requirements part of the Docker Image, which supports Poetry projects, but poetry is not installed in the image?

I've tried installing poetry in a pre execution script, and I still get the same error. Can the serverless-deploy pipe's Dockerfile include the installation of Poetry?

Thanks

Accepted answers

Oleksandr Kyrdan

@Sam Mahr

Thank you for your question!

The pipe's base docker image contains only base dependencies to keep the image thin.

There are many variety of dependencies all around the users projects.

If you need extra dependencies according to your project, pipe supports preexecution script.

Additionally, check if the version of requirements and runtime corresponds to platform dependency version in Dockerfile.

Also, could you provide your bitbucket-pipelines configuration, serverless configuration and preexecution script.

Best regards,
Oleksandr Kyrdan

All comments

Oleksandr Kyrdan

@Sam Mahr

Thank you for your question!

The pipe's base docker image contains only base dependencies to keep the image thin.

There are many variety of dependencies all around the users projects.

If you need extra dependencies according to your project, pipe supports preexecution script.

Additionally, check if the version of requirements and runtime corresponds to platform dependency version in Dockerfile.

Also, could you provide your bitbucket-pipelines configuration, serverless configuration and preexecution script.

Best regards,
Oleksandr Kyrdan

Sam Mahr

Thanks, @Oleksandr Kyrdan. Here's the files

apt-get update
apt-get install -y git

# START SSH setup
INJECTED_SSH_CONFIG_DIR="/opt/atlassian/pipelines/agent/ssh"
# The default ssh key with open perms readable by alt uids
IDENTITY_FILE="${INJECTED_SSH_CONFIG_DIR}/id_rsa_tmp"
# The default known_hosts file
KNOWN_HOSTS_FILE="${INJECTED_SSH_CONFIG_DIR}/known_hosts"

mkdir -p ~/.ssh || debug "adding ssh keys to existing ~/.ssh"
touch ~/.ssh/authorized_keys

# If given, use SSH_KEY, otherwise check if the default is configured and use it
if [ "${SSH_KEY}" != "" ]; then
   debug "Using passed SSH_KEY"
   (umask  077 ; echo ${SSH_KEY} | base64 -d > ~/.ssh/pipelines_id)
elif [ ! -f ${IDENTITY_FILE} ]; then
   error "No default SSH key configured in Pipelines."
   exit 1
else
   debug "Using default ssh key"
   cp ${IDENTITY_FILE} ~/.ssh/pipelines_id
fi

if [ ! -f ${KNOWN_HOSTS_FILE} ]; then
    error "No SSH known_hosts configured in Pipelines."
    exit 2
fi

cat ${KNOWN_HOSTS_FILE} >> ~/.ssh/known_hosts
if [ -f ~/.ssh/config ]; then
    debug "Appending to existing ~/.ssh/config file"
fi
echo "IdentityFile ~/.ssh/pipelines_id" >> ~/.ssh/config
chmod -R go-rwx ~/.ssh/
# END SSH setup

curl -sSL https://install.python-poetry.org | python3 -
export PATH="$HOME/.local/bin:$PATH"
poetry --version
poetry config http-basic.artifact aws "$TOKEN"

npm install --save -g serverless-step-functions

pre execution script

service: ServerlessAppTemplate
frameworkVersion: ^3.38.0
package:
  individually: true
patterns:
    - '!config/**'
    - '!lambda_functions/**'
    - '!*.json'
    - '!*.md'
    - '!*.txt'
    - '!.gitignore'
    - '!pytest.ini'
    - '!tests/**'
    - '!*.yml'
    - '!*.yaml'
    - '!Makefile'
    - '!poetry.lock'
    - '!pyproject.toml'

provider:
  name: aws
runtime: python3.10
httpApi:
    cors: true
stage: ${opt:stage, "dev"}
region: ${opt:region, "us-east-2"}
deploymentBucket: 
    name: [redacted]-${self:provider.region}
deploymentPrefix: Serverless
environment:
    REGION: ${self:provider.region}
STAGE: ${self:provider.stage}
POWERTOOLS_SERVICE_NAME: template
LOG_LEVEL: INFO

functions: ${file(config/functions.yml)}

resources: ${file(config/resources.yml)}

stepFunctions: ${file(config/state_machines.yml)}

plugins:
  - serverless-python-requirements
- serverless-step-functions

custom:
  powertoolsLayer: arn:aws:lambda:${opt:region}:017000801446:layer:AWSLambdaPowertoolsPythonV2:67
pythonRequirements:
layer: true
slim: true
strip: false
    usePoetry: true

Serverless config with explicit usePoetry

image: python:3.10

definitions:
  caches:
    poetry-deps:
      key:
        files:
          - poetry.lock
path: .venv
steps:
    - step: &get-token-step
        name: Get CodeArtifact Token
caches:
          - pip
- poetry-deps
image: atlassian/pipelines-awscli
script:
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=us-east-2
- TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain [redacted] --query authorizationToken --output text)
- echo "export TOKEN=$TOKEN" >> set_env.sh
artifacts:
          - set_env.sh
- step: &audit-step
        name: Audit Packages
caches:
          - docker
- pip
- poetry-deps
script:
          - source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry self add poetry-audit-plugin
- poetry audit
- step: &test-step
        name: Test
caches:
          - docker
- pip
- poetry-deps
script:
          - source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run python3 -m pytest --cov=./ --cov-report xml
    - step: &lint-step
        name: Lint code
caches:
          - docker
- pip
- poetry-deps
script:
          - source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run ruff check

- step: &deploy-functions-step
        name: Deploy Functions
caches:
          - docker
- node
- pip
script:
          - &deploy-functions-script
            pipe: atlassian/serverless-deploy:1.5.0
variables: &deploy-functions-variables
              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
EXTRA_ARGS: "-s $STAGE --region $REGION"
              DEBUG: "true"
              TOKEN: $TOKEN
PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

pipelines:
  default:
    - step: *get-token-step
    - parallel:
        - step: *audit-step
        - step: *test-step
        - step: *lint-step

  branches:
    main:
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          <<: *deploy-functions-step
          name: Deploy to stage Dev, region us-east-2
deployment: dev-us-east-2
script:
            - source set_env.sh
- <<: *deploy-functions-script
              variables:
                <<: *deploy-functions-variables
tags:
    'v*':
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          <<: *deploy-functions-step
          name: Deploy to stage Prod, region us-east-2
deployment: prod-us-east-2
trigger: manual
script:
            - source set_env.sh
- <<: *deploy-functions-script
              variables:
                <<: *deploy-functions-variables

pipelines file

image: python:3.10

definitions:
  caches:
    poetry-deps:
      key:
        files:
          - poetry.lock
path: .venv
steps:
    - step: &get-token-step
        name: Get CodeArtifact Token
caches:
          - pip
- poetry-deps
image: atlassian/pipelines-awscli
script:
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=us-east-2
- TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain patronpoint-artifact --query authorizationToken --output text)
- echo "export TOKEN=$TOKEN" >> set_env.sh
artifacts:
          - set_env.sh
- step: &audit-step
        name: Audit Packages
caches:
          - docker
- pip
- poetry-deps
script:
          - source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry self add poetry-audit-plugin
- poetry audit
- step: &test-step
        name: Test
caches:
          - docker
- pip
- poetry-deps
script:
          - source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run python3 -m pytest --cov=./ --cov-report xml
# Install deepsource CLI
          # - curl https://deepsource.io/cli | sh
          # From the root directory, run the report coverage command
          # upload coverage report to deepsource
          # - ./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml
    - step: &lint-step
        name: Lint code
caches:
          - docker
- pip
- poetry-deps
script:
          - source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run ruff check

- step: &deploy-functions-step
        name: Deploy Functions
caches:
          - docker
- node
- pip
script:
          - &deploy-functions-script
            pipe: atlassian/serverless-deploy:1.5.0
variables: &deploy-functions-variables
              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
EXTRA_ARGS: "-s $STAGE --region $REGION"
              DEBUG: "true"
              TOKEN: $TOKEN
PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

pipelines:
  default:
    - step: *get-token-step
    - parallel:
        - step: *audit-step
        - step: *test-step
        - step: *lint-step

  branches:
    main:
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          name: Deploy to stage Dev, region us-east-2
caches:
            - docker
- node
- pip
deployment: dev-us-east-2
script:
            - source set_env.sh
- pipe: atlassian/serverless-deploy:1.5.0
variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
EXTRA_ARGS: "-s $STAGE --region $REGION"
                DEBUG: "true"
                TOKEN: $TOKEN
PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

Same issue with this pipelines file as well

Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
× Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
✖ Deployment failed

Sam Mahr

@Oleksandr Kyrdan Following up

Oleksandr Kyrdan

Hi @Sam Mahr

Could you try the new dev version of the pipe with Poetry in the pipe's base docker image before the release and share feedback:

- pipe: docker://bitbucketpipelines/serverless-deploy:1.5.0.163-dev

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

bitbucket

bitbucket-cloud

bitbucket-pipelines

serverless

pipes