I have a serverless app that uses Poetry for package management.
I am using bitbucketpipelines/serverless-deploy:1.5.0 to deploy my serverless app. When I try to deploy, I get this error:
Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
× Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
Why is serverless-python-requirements part of the Docker Image, which supports Poetry projects, but poetry is not installed in the image?
I've tried installing poetry in a pre execution script, and I still get the same error. Can the serverless-deploy pipe's Dockerfile include the installation of Poetry?
Thanks
Thank you for your question!
The pipe's base docker image contains only base dependencies to keep the image thin.
There are many variety of dependencies all around the users projects.
If you need extra dependencies according to your project, pipe supports preexecution script.
Additionally, check if the version of requirements and runtime corresponds to platform dependency version in Dockerfile.
Also, could you provide your bitbucket-pipelines configuration, serverless configuration and preexecution script.
Best regards,
Oleksandr Kyrdan
Thanks, @Oleksandr Kyrdan. Here's the files
apt-get update
apt-get install -y git
# START SSH setup
INJECTED_SSH_CONFIG_DIR="/opt/atlassian/pipelines/agent/ssh"
# The default ssh key with open perms readable by alt uids
IDENTITY_FILE="${INJECTED_SSH_CONFIG_DIR}/id_rsa_tmp"
# The default known_hosts file
KNOWN_HOSTS_FILE="${INJECTED_SSH_CONFIG_DIR}/known_hosts"
mkdir -p ~/.ssh || debug "adding ssh keys to existing ~/.ssh"
touch ~/.ssh/authorized_keys
# If given, use SSH_KEY, otherwise check if the default is configured and use it
if [ "${SSH_KEY}" != "" ]; then
debug "Using passed SSH_KEY"
(umask 077 ; echo ${SSH_KEY} | base64 -d > ~/.ssh/pipelines_id)
elif [ ! -f ${IDENTITY_FILE} ]; then
error "No default SSH key configured in Pipelines."
exit 1
else
debug "Using default ssh key"
cp ${IDENTITY_FILE} ~/.ssh/pipelines_id
fi
if [ ! -f ${KNOWN_HOSTS_FILE} ]; then
error "No SSH known_hosts configured in Pipelines."
exit 2
fi
cat ${KNOWN_HOSTS_FILE} >> ~/.ssh/known_hosts
if [ -f ~/.ssh/config ]; then
debug "Appending to existing ~/.ssh/config file"
fi
echo "IdentityFile ~/.ssh/pipelines_id" >> ~/.ssh/config
chmod -R go-rwx ~/.ssh/
# END SSH setup
curl -sSL https://install.python-poetry.org | python3 -
export PATH="$HOME/.local/bin:$PATH"
poetry --version
poetry config http-basic.artifact aws "$TOKEN"
npm install --save -g serverless-step-functions
pre execution script
service: ServerlessAppTemplate
frameworkVersion: ^3.38.0
package:
individually: true
patterns:
- '!config/**'
- '!lambda_functions/**'
- '!*.json'
- '!*.md'
- '!*.txt'
- '!.gitignore'
- '!pytest.ini'
- '!tests/**'
- '!*.yml'
- '!*.yaml'
- '!Makefile'
- '!poetry.lock'
- '!pyproject.toml'
provider:
name: aws
runtime: python3.10
httpApi:
cors: true
stage: ${opt:stage, "dev"}
region: ${opt:region, "us-east-2"}
deploymentBucket:
name: [redacted]-${self:provider.region}
deploymentPrefix: Serverless
environment:
REGION: ${self:provider.region}
STAGE: ${self:provider.stage}
POWERTOOLS_SERVICE_NAME: template
LOG_LEVEL: INFO
functions: ${file(config/functions.yml)}
resources: ${file(config/resources.yml)}
stepFunctions: ${file(config/state_machines.yml)}
plugins:
- serverless-python-requirements
- serverless-step-functions
custom:
powertoolsLayer: arn:aws:lambda:${opt:region}:017000801446:layer:AWSLambdaPowertoolsPythonV2:67
pythonRequirements:
layer: true
slim: true
strip: false
usePoetry: true
Serverless config with explicit usePoetry
image: python:3.10
definitions:
caches:
poetry-deps:
key:
files:
- poetry.lock
path: .venv
steps:
- step: &get-token-step
name: Get CodeArtifact Token
caches:
- pip
- poetry-deps
image: atlassian/pipelines-awscli
script:
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=us-east-2
- TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain [redacted] --query authorizationToken --output text)
- echo "export TOKEN=$TOKEN" >> set_env.sh
artifacts:
- set_env.sh
- step: &audit-step
name: Audit Packages
caches:
- docker
- pip
- poetry-deps
script:
- source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry self add poetry-audit-plugin
- poetry audit
- step: &test-step
name: Test
caches:
- docker
- pip
- poetry-deps
script:
- source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run python3 -m pytest --cov=./ --cov-report xml
- step: &lint-step
name: Lint code
caches:
- docker
- pip
- poetry-deps
script:
- source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run ruff check
- step: &deploy-functions-step
name: Deploy Functions
caches:
- docker
- node
- pip
script:
- &deploy-functions-script
pipe: atlassian/serverless-deploy:1.5.0
variables: &deploy-functions-variables
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
EXTRA_ARGS: "-s $STAGE --region $REGION"
DEBUG: "true"
TOKEN: $TOKEN
PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'
pipelines:
default:
- step: *get-token-step
- parallel:
- step: *audit-step
- step: *test-step
- step: *lint-step
branches:
main:
- step: *get-token-step
- parallel:
- step: *audit-step
- step: *test-step
- step: *lint-step
- step:
<<: *deploy-functions-step
name: Deploy to stage Dev, region us-east-2
deployment: dev-us-east-2
script:
- source set_env.sh
- <<: *deploy-functions-script
variables:
<<: *deploy-functions-variables
tags:
'v*':
- step: *get-token-step
- parallel:
- step: *audit-step
- step: *test-step
- step: *lint-step
- step:
<<: *deploy-functions-step
name: Deploy to stage Prod, region us-east-2
deployment: prod-us-east-2
trigger: manual
script:
- source set_env.sh
- <<: *deploy-functions-script
variables:
<<: *deploy-functions-variables
pipelines file
image: python:3.10
definitions:
caches:
poetry-deps:
key:
files:
- poetry.lock
path: .venv
steps:
- step: &get-token-step
name: Get CodeArtifact Token
caches:
- pip
- poetry-deps
image: atlassian/pipelines-awscli
script:
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=us-east-2
- TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain patronpoint-artifact --query authorizationToken --output text)
- echo "export TOKEN=$TOKEN" >> set_env.sh
artifacts:
- set_env.sh
- step: &audit-step
name: Audit Packages
caches:
- docker
- pip
- poetry-deps
script:
- source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry self add poetry-audit-plugin
- poetry audit
- step: &test-step
name: Test
caches:
- docker
- pip
- poetry-deps
script:
- source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run python3 -m pytest --cov=./ --cov-report xml
# Install deepsource CLI
# - curl https://deepsource.io/cli | sh
# From the root directory, run the report coverage command
# upload coverage report to deepsource
# - ./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml
- step: &lint-step
name: Lint code
caches:
- docker
- pip
- poetry-deps
script:
- source set_env.sh
- pipe: atlassian/poetry-cli-setup:0.2.0
- POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
- poetry config http-basic.artifact aws $TOKEN
- poetry install
- poetry run ruff check
- step: &deploy-functions-step
name: Deploy Functions
caches:
- docker
- node
- pip
script:
- &deploy-functions-script
pipe: atlassian/serverless-deploy:1.5.0
variables: &deploy-functions-variables
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
EXTRA_ARGS: "-s $STAGE --region $REGION"
DEBUG: "true"
TOKEN: $TOKEN
PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'
pipelines:
default:
- step: *get-token-step
- parallel:
- step: *audit-step
- step: *test-step
- step: *lint-step
branches:
main:
- step: *get-token-step
- parallel:
- step: *audit-step
- step: *test-step
- step: *lint-step
- step:
name: Deploy to stage Dev, region us-east-2
caches:
- docker
- node
- pip
deployment: dev-us-east-2
script:
- source set_env.sh
- pipe: atlassian/serverless-deploy:1.5.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
EXTRA_ARGS: "-s $STAGE --region $REGION"
DEBUG: "true"
TOKEN: $TOKEN
PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'
Same issue with this pipelines file as well
Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
× Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
✖ Deployment failed :(
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Sam Mahr
Could you try the new dev version of the pipe with Poetry in the pipe's base docker image before the release and share feedback:
- pipe: docker://bitbucketpipelines/serverless-deploy:1.5.0.163-dev
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Retrieving Poetry metadata
The latest version (1.8.3) is already installed.
Poetry (version 1.8.3)
eyJ2ZXIiOjEsImlzdSI6MTcxNTgyMjM2MiwiZW5jIjoiQTEyOEdDTSIsInRhZyI6IjQ1bzVrZ2J4b3BleXNjdU04TW9Qa0EiLCJleHAiOjE3MTU4NjU1NjIsImFsZyI6IkExMjhHQ01LVyIsIml2IjoiSXVveEsxT1dpREtyNW1qRiJ9.6uRiS-LVWp3B2eNfDi31LA.DT-uf1zhaXMHhYFJ.iB0zmDgkc0ykdf3bjA9iQAJL_5Xj9C2_Td1Oppes2xF9Krp6iTK9ffoYf5oqduOC8QtMMpET_4svNkXvLnfoi2iTBJW_W76j2rsWiEw-nPueM0-ff_J0ptZ4nc3DOJGYqfW6vtsfnlBkL3MfyyqO4ts7k01IFx7SURxHP8eTGXHFlRATtAwTgWNbwv4W4igCzirxYYNqcWUfcRkQthZrRKypOd1jMJ20_wJLpkwD5wPZl9BTWgWHZxTy9xXdHLXjldWK24CLe2X62NcpKTHcWol23BrTrecD4a8EqvAmjxURyaLrbsVI_NVQmrSX26z58WKpQ2x3DcVdDKvwUC5HcmhOUrD5-AF-4c-CaUNO7qVurjzCvmcNFCL5n0usSGdmILHR0864sf7E_INYaozyt8-BvMsyCHI154teaDCYUIoSNa7bMQRWwnBZLddpQijyGT5IYUmb2i6gVyH3EmV34dgVqksI9yDngrI2Ea5Lopasb-WBYliPK3srOZF4r7xTaS_vG-SMjAXTI1X_o9-WjhyYK4k1SrMrXMu3z_fFhcVeFI_pbDXBE13UaL7swr9RwxAjuS7AEtOnFuxG1PnsonrcH508sfnXJEuI0-gx8nOOfKrtkY6ZG5ed2oV4maMhGhX2inIaBwPF6L5etz4Nk0t2NEqJJ-AVo_gnGup6vIu8JhRVk1HYBjoHjTPKW-6eOlyluQ_l6ElwoAdJm3H3RoEGn8mbTH4r9ol8HsBW5KLJg0D5ccwBfR3EQ40r3EmVmp0V3bJGlIRYgTU_H1hGI6gxqqKAZHtNzY6ttTd-bJzQCtPttMW1bwCb5LH68LuDKtSKQCZvr4X_aAcrButnk4svVD2nCIo31hP9qSVEBo_KXsMgaIYuXDiX10cz7zGaJhTY4Wi32_Rcd_5jNYXSgukqKio.befgFQ-ZgjR0mZZnWdkYVQ
added 570 packages in 21s
113 packages are looking for funding
run `npm fund` for details
INFO: Deploying your serverless application...
DEBUG: serverless deploy -c serverless.yml -s dev --region us-east-2 --verbose
Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
Parsed requirements.txt from pyproject.toml in /opt/atlassian/pipelines/agent/build/.serverless/requirements.txt
Installing requirements from "/root/.cache/serverless-python-requirements/1f0dd62c2bede82c87a91b6c441f0d0940af59d12684e386817a7ff2757d10a6_x86_64_slspyc/requirements.txt"
Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
× Stack ServerlessAppTemplate-dev failed to deploy (1s)
Environment: linux, node 20.13.1, framework 3.38.0, plugin 7.2.3, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn python3.10 ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
✖ Deployment failed :(
Hmm, it looks like there's still an issue.
The pre execution script tries to install poetry, but it's already installed in this docker image, but still getting this issue: "Error: spawn python3.10 ENOENT"
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Sam Mahr please, pay attention that pipe's docker image based on python:3.11-slim. Looks like you try to spawn python3.10 instead.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Ah, I did not look at the image. I've changed everything to 3.11 and now it is working and deployed! @Oleksandr Kyrdan
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Nice! Thanks for your feedback! 👍
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Sam Mahr
A new version of serverless-deploy pipe is released:
- pipe: atlassian/serverless-deploy:2.0.0
changes:
2.0.0
major: Breaking changes. Refactored pipe to support better auth logic and major providers AWS, GCP, AZURE.
minor: Add OIDC support for AWS provider. Add support for AWS_OIDC_ROLE_ARN variable.
minor: Add Poetry to the pipe's base docker image as an alternative tool for dependency management and packaging in Python.
minor: Improved support for Azure cloud. Add serverless-azure-functions to the default environment.
minor: Update base NodeJS version to nodejs20.
minor: Update pipe's base docker image to python:3.11-slim.
patch: Internal maintenance: bump project's dependencies to the latest.
Best regards,
Oleksandr Kyrdan
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.