Pipeline error: yaml.scanner.ScannerError: mapping values are not allowed

[Srinivas Nekkalapudi]

Sometimes pipeline is successful and a few times it is failing.

I assume the dynamic BITBUCKET_BRANCH name might be the issue. 

INFO: Successfully updated the kube config.

Traceback (most recent call last):

  File "/pipe.py", line 47, in <module>

    pipe.run()

  File "/root/.local/lib/python3.8/site-packages/kubectl_run/pipe.py", line 157, in run

    self.handle_apply()

  File "/root/.local/lib/python3.8/site-packages/kubectl_run/pipe.py", line 114, in handle_apply

    self.update_labels_in_metadata(template_file, labels)

  File "/root/.local/lib/python3.8/site-packages/kubectl_run/pipe.py", line 30, in update_labels_in_metadata

    yamls = list(yaml.safe_load_all(template_file.read()))

  File "/root/.local/lib/python3.8/site-packages/yaml/__init__.py", line 130, in load_all

    yield loader.get_data()

  File "/root/.local/lib/python3.8/site-packages/yaml/constructor.py", line 45, in get_data

    return self.construct_document(self.get_node())

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 27, in get_node

    return self.compose_document()

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document

    node = self.compose_node(None, None)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

    node = self.compose_mapping_node(anchor)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

    item_value = self.compose_node(node, item_key)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

    node = self.compose_mapping_node(anchor)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

    item_value = self.compose_node(node, item_key)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

    node = self.compose_mapping_node(anchor)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

    item_value = self.compose_node(node, item_key)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

    node = self.compose_mapping_node(anchor)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

    item_value = self.compose_node(node, item_key)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 82, in compose_node

    node = self.compose_sequence_node(anchor)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 111, in compose_sequence_node

    node.value.append(self.compose_node(node, index))

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

    node = self.compose_mapping_node(anchor)

  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 127, in compose_mapping_node

    while not self.check_event(MappingEndEvent):

  File "/root/.local/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event

    self.current_event = self.state()

  File "/root/.local/lib/python3.8/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key

    if self.check_token(KeyToken):

  File "/root/.local/lib/python3.8/site-packages/yaml/scanner.py", line 116, in check_token

    self.fetch_more_tokens()

  File "/root/.local/lib/python3.8/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens

    return self.fetch_value()

  File "/root/.local/lib/python3.8/site-packages/yaml/scanner.py", line 577, in fetch_value

    raise ScannerError(None, None,

yaml.scanner.ScannerError: mapping values are not allowed here

  in "<unicode string>", line 30, column 79:

     ... -1.amazonaws.com/aupris-frontend:

bitbucker-pipeline file

image: atlassian/default-image:2

options:     

  docker: true

  size: 2x

pipelines:

  tags:

    release-*:

      - step:

          name: Build docker image

          services:

            - docker

          script:

          # build the image

          - docker build -t aupris-frontend .

          # use the pipe to push the image to AWS ECR

          - pipe: atlassian/aws-ecr-push-image:1.4.2

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

              IMAGE_NAME: aupris-frontend

              TAGS: '${BITBUCKET_BRANCH}'

          - apt-get update && apt-get install gettext-base

          - envsubst < aupris-frontend.tpl.yaml > aupris-frontend.yaml

          - sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-frontend.yaml

          - pipe: "atlassian/aws-eks-kubectl-run:2.2.0"

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: XXXXX

              CLUSTER_NAME: XXXXX

              KUBECTL_COMMAND: "apply"

              RESOURCE_PATH: "aupris-frontend.yaml"

              DEBUG: "true"

definitions:

  services:

    docker:

      memory: 7128

yaml file

apiVersion: v1

kind: Service

metadata:

  name: aupris-frontend

spec:

  # type: LoadBalancer

  selector:

    app: aupris-frontend

  ports:

    - name: http

      port: 80

      targetPort: 80

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: aupris-frontend

spec:

  selector:

    matchLabels:

      app: aupris-frontend

  template:

    metadata:

      labels:

        app: aupris-frontend

    spec:

      terminationGracePeriodSeconds: 30

      containers:

        - name: aupris-frontend

          image: XXXXXXXXX:${BITBUCKET_BRANCH}

          imagePullPolicy: "Always"

          env:

            - name: API_URL

              value: api/

            - name: FOR_GODS_SAKE_PLEASE_REDEPLOY

              value: "THIS_STRING_IS_REPLACED_DURING_BUILD"

          resources:

            limits:

              memory: "128Mi"

              cpu: "100m"

          ports:

            - containerPort: 80

[Igor Stoyanov]

@Srinivas Nekkalapudi  hi.

You should not use dynamical  variables inside your aupris-frontend yaml Kubernetes manifest file. And As @Oday Rafeh mentioned, you could create template instead, fill it with needed variables and create according manifest. Also, optionally, you could validate your generated manifest.

Regards, Igor

[Erez Maadani]

Hi @Srinivas Nekkalapudi ,

1. It seems you have different python versions. Your original post shows python3.8 while the screenshot of the "pip install.." shows python2.7. You might need a different runner image for your pipeline.
2. 2+3 - Every step of the pipeline is running in different container, so by default, env parameters and files are not shared. You might need to define a cache or artifacts to pass files between steps. Once you can share files, you can also share env parameters using files, just dump and read to/from a shard file. This link might help: https://community.atlassian.com/t5/Bitbucket-questions/caches-vs-artifacts/qaq-p/864904

[Oday Rafeh]

@Srinivas Nekkalapudi 

The error is due to a problem in your YAML file. To fix it:

Check for missing colons.
Ensure proper indentation.
Quote strings with special characters.
Fix unintended line breaks.
Review the YAML file, fix the issues, and retry running the pipeline.

IF this didn't help please provide the YAML file content so I can help more. 

[Srinivas Nekkalapudi]

@Oday Rafeh Thanks for the quick response. I have edited my post and updated yaml file. 

I wonder why my pipeline is successful few time with the same configuration and failing few other times.

I could not reply to your answer so editing my old comment.

I came across another error with your solution:

+ yamllint aupris-server.yaml

Traceback (most recent call last):

  File "/usr/bin/yamllint", line 5, in <module>

    from pkg_resources import load_entry_point

ImportError: No module named 'pkg_resources'

---

@Oday Rafeh It still did not solve the issue. following is the screenshot:

[Oday Rafeh]

I noticed you have the image URL redacted with XXXXXXXXX. If the actual image URL in your YAML file contains any special characters, like colons, you need to wrap the value in quotes.

The updated section with quotes:

 containers:
- name: aupris-frontend
image: "XXXXXXXXX:${BITBUCKET_BRANCH}"
imagePullPolicy: "Always"

[Oday Rafeh]

To address the intermittent pipeline failures related to the dynamic BITBUCKET_BRANCH name, you can modify your bitbucket-pipelines. yml file by adding a step to log the branch name, validate the YAML file using yamllint, and replace special characters in the branch name with underscores.

Updated bitbucket-pipelines. yml:

image: atlassian/default-image:2

options:
docker: true
size: 2x

pipelines:
tags:
release-*:
- step:
name: Prepare and validate
script:
- echo "BITBUCKET_BRANCH: ${BITBUCKET_BRANCH}"
- apt-get update && apt-get install -y gettext-base yamllint
- export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"
- echo "BITBUCKET_BRANCH_SAFE: ${BITBUCKET_BRANCH_SAFE}"
- envsubst < aupris-frontend.tpl.yaml > aupris-frontend.yaml
- yamllint aupris-frontend.yaml
- step:
name: Build docker image
services:
- docker
script:
# build the image
- docker build -t aupris-frontend .
# use the pipe to push the image to AWS ECR
- pipe: atlassian/aws-ecr-push-image:1.4.2
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
IMAGE_NAME: aupris-frontend
TAGS: '${BITBUCKET_BRANCH_SAFE}'
- sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-frontend.yaml
- pipe: "atlassian/aws-eks-kubectl-run:2.2.0"
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: XXXXX
CLUSTER_NAME: XXXXX
KUBECTL_COMMAND: "apply"
RESOURCE_PATH: "aupris-frontend.yaml"
DEBUG: "true"

definitions:
services:
docker:
memory: 7128

I added new step called Prepare and validate". This step logs the branch name, installs yamllint, and creates a sanitized version of the branch name called BITBUCKET_BRANCH_SAFE. The branch name is then used in the subsequent steps for building the Docker image and deploying it.

Please update the aupris-frontend. tpl. yaml file to use ${BITBUCKET_BRANCH_SAFE} instead of ${BITBUCKET_BRANCH} for the image name:

# ... rest of the file
containers:
- name: aupris-frontend
image: "XXXXXXXXX:${BITBUCKET_BRANCH_SAFE}"
imagePullPolicy: "Always"
# ... rest of the file

[Oday Rafeh]

Hi @Srinivas Nekkalapudi , It seems that the 'pkg_resources' module is missing from your environment. You can install it by adding this command to your pipeline script:

pip install setuptools

The fully updated "Prepare and validate" step should be like this : 

- step:
name: Prepare and validate
script:
- echo "BITBUCKET_BRANCH: ${BITBUCKET_BRANCH}"
- apt-get update && apt-get install -y gettext-base yamllint
- pip install setuptools
- export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"
- echo "BITBUCKET_BRANCH_SAFE: ${BITBUCKET_BRANCH_SAFE}"
- envsubst < aupris-frontend.tpl.yaml > aupris-frontend.yaml
- yamllint aupris-frontend.yaml

[Srinivas Nekkalapudi]

@Oday Rafeh Please have a look at the screenshot.

[Oday Rafeh]

@Srinivas Nekkalapudi 

Try this please should work based on your screenshot : 

sudo apt-get install python-setuptools

Prepare and validate" step should be like this now :

- step:
name: Install dependencies
script:
- apt-get update && apt-get install -y python-setuptools

[Srinivas Nekkalapudi]

@Oday Rafeh Even with that it did not work

Found one more issue with the pipeline code: 

the file could not be found.

BTW I skipped yamllint line to go further into other steps.

here is the bitbucket pipeline code:

image: atlassian/default-image:2

pipelines:

  tags:

    release-*:

      - step:

          name: Prepare and validate

          script:

            - echo "BITBUCKET_BRANCH  ${BITBUCKET_BRANCH}"

            - apt-get update && apt-get install -y python-setuptools gettext-base yamllint

            - export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"

            - echo "BITBUCKET_BRANCH_SAFE ${BITBUCKET_BRANCH_SAFE}"

            - envsubst < aupris-server.tpl.yaml > aupris-server.yaml

            - yamllint aupris-server.yaml

      - step:

          name: Build docker image and push to ECR

          services:

            - docker

          script:

          # build the image

          - docker build -t aupris-server .

          # use the pipe to push the image to AWS ECR

          - pipe: atlassian/aws-ecr-push-image:1.4.2

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

              IMAGE_NAME: aupris-server

              TAGS: '${BITBUCKET_BRANCH_SAFE}'

      - step:

          name: Releasing to kubernetes via EKS

          script: 

          - sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-server.yaml

          - pipe: "atlassian/aws-eks-kubectl-run:2.2.0"

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

              CLUSTER_NAME: "julius66"

              KUBECTL_COMMAND: "apply"

              RESOURCE_PATH: "aupris-server.yaml"

              DEBUG: "true"

[Oday Rafeh]

@Srinivas Nekkalapudi 

The error "sed: can't read aupris-server. yaml: No such file or directory" indicates that the file 'aupris-server. yaml' is not found in the current directory.

You should ensure that the 'aupris-server. tpl. yaml' file is present in the same directory as the bitbucket-pipelines. yml file, and that it has been correctly defined to generate the 'aupris-server. yaml' file.

You can also check the output of the 'Prepare and validate' step to see if there were any errors related to the creation of the aupris-server. yaml file.

[Srinivas Nekkalapudi]

@Oday Rafeh 

aupris-server.yaml and bitbucket-pipelines. yml are in the same directory.

there are no errors in  'Prepare and validate' step other than 

ImportError: No module named 'pkg_resources'

Is it because of the STEPS? 

when i do echo "BITBUCKET_BRANCH_SAFE ${BITBUCKET_BRANCH_SAFE}" in step "Releasing to Kubernetes via EKS" it gives me an output like "BITBUCKET_BRANCH_SAFE   ".

[Oday Rafeh]

@Srinivas Nekkalapudi 

The "BITBUCKET_BRANCH_SAFE" variable seems to be empty in the "Releasing to Kubernetes via EKS" step, which could be causing the issue with the sed command. One possible reason for this could be that the environment variable is not being passed to the subsequent step properly.

To troubleshoot this issue, you can try adding a debug statement in the "Build docker image and push to ECR" step to confirm that the "BITBUCKET_BRANCH_SAFE" variable is being properly set, If you want help with that let me know I am here to help 

[Srinivas Nekkalapudi]

@Oday Rafeh It's empty in step "Build docker image and push to ECR" as well

[Oday Rafeh]

@Srinivas Nekkalapudi 

the issue is with the "Build docker image and push to ECR" step. The docker build command seems to be missing the context path and is not able to find the Dockerfile.

Please update the pipeline code

image: atlassian/default-image:2

pipelines:

tags:

release-*:

- step:

name: Prepare and validate

script:

- echo "BITBUCKET_BRANCH ${BITBUCKET_BRANCH}"

- apt-get update && apt-get install -y python-setuptools gettext-base yamllint

- export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"

- echo "BITBUCKET_BRANCH_SAFE ${BITBUCKET_BRANCH_SAFE}"

- envsubst < aupris-server.tpl.yaml > aupris-server.yaml

- yamllint aupris-server.yaml

- step:

name: Build docker image and push to ECR

services:

- docker

script:

# build the image

- docker build -t aupris-server -f Dockerfile .

# use the pipe to push the image to AWS ECR

- pipe: atlassian/aws-ecr-push-image:1.4.2

variables:

AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

IMAGE_NAME: aupris-server

TAGS: '${BITBUCKET_BRANCH_SAFE}'

- step:

name: Releasing to kubernetes via EKS

script:

- sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-server.yaml

- pipe: "atlassian/aws-eks-kubectl-run:2.2.0"

variables:

AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

CLUSTER_NAME: "julius66"

KUBECTL_COMMAND: "apply"

RESOURCE_PATH: "aupris-server.yaml"

DEBUG: "true"

 I added the "-f Dockerfile" option to the docker build command to specify the location of the Dockerfile, which should be in the same directory as the pipeline file. This should resolve the issue with the empty output

[Srinivas Nekkalapudi]

@Oday Rafeh I don't think there is anything wrong with the context path.

[Oday Rafeh]

@Srinivas Nekkalapudi 

Correct, The output from the pipeline suggests that the Docker image is being built successfully and the BITBUCKET_BRANCH_SAFE variable is being printed correctly. The pipeline then goes on to push the Docker image to AWS ECR using the atlassian/aws-ecr-push-image pipe. Without more information, it's difficult to determine if there are any issues with this step.

[Oday Rafeh]

@Srinivas Nekkalapudi , 

I was thinking a little bit out of the box and I tried to validate your YAML code

here its :

 The error seems to be related to the YAML syntax of the aupris-frontend. yaml file. The error message specifically mentions that "mapping values are not allowed here". This error usually occurs when there is a syntax error in the YAML file.

This error typically occurs when there are multiple YAML documents in a single file. Each YAML file should contain only one document. Please check your YAML file and make sure that it contains only one document. If you have multiple documents, you can separate them by adding three hyphens (---) between each document.

[Oday Rafeh]

So lets Try this file instead : 

---
apiVersion: v1
kind: Service
metadata:
name: aupris-frontend
spec:
selector:
app: aupris-frontend
ports:
- name: http
port: 80
targetPort: 80

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: aupris-frontend
spec:
selector:
matchLabels:
app: aupris-frontend
template:
metadata:
labels:
app: aupris-frontend
spec:
terminationGracePeriodSeconds: 30
containers:
- name: aupris-frontend
image: XXXXXXXXX:${BITBUCKET_BRANCH}
imagePullPolicy: Always
env:
- name: API_URL
value: api/
- name: FOR_GODS_SAKE_PLEASE_REDEPLOY
value: THIS_STRING_IS_REPLACED_DURING_BUILD
resources:
limits:
memory: 128Mi
cpu: 100m
ports:
- containerPort: 80

[Srinivas Nekkalapudi]

@Oday Rafeh HI, If you see my original post I have a single file with two documents separated by (---).

so there are no changes to make I believe.

To summarise, the following are the issues I have:

• ImportError: No module named 'pkg_resources'
• Could not access BITBUCKET_BRANCH_SAFE in steps 2 and 3.
• Newly created files(e.g: sed -ie) could not be accessed in steps 2 and 3.