The deployment environment 'test' in your bitbucket-pipelines.yml file is invalid.

[Johnnie Kearse]

Hello,

According to Bitbucket, my bitbucket-pipelines.yml file is invalid due to deployment environment 'test' being invalid. Screenshot below:

I checked the validator and it said that my code is valid, so I'm confused as to what I am doing wrong.

My .yml file is as follows:

definitions:
  steps:
    - step: &step-test
        name: "Test"
        image: golang:1.11-stretch
        deployment: test
        script:
          - export GOPATH="$HOME/go"
          - export PATH="$PATH:$GOPATH/bin"
          - go get -u golang.org/x/lint/golint
          - go mod vendor
          - golint -set_exit_status $(go list ./... | grep -v /vendor/)
          # Uncomment these test steps when there are test file available
          # - go test -short $(go list ./... | grep -v /vendor/)
          # - go test -race -short $(go list ./... | grep -v /vendor/)
        artifacts:
          - vendor/**
    - step: &step-build
        name: "Build"
        image: golang:1.11-stretch
        deployment: test
        script:
          - go build
    - step: &step-deploy
        # set GCLOUD_PROJECT environment variable to your project ID
        # set GCLOUD_API_KEYFILE environment variable to keyfile as described here (but we aren't base64-encoding the file): https://confluence.atlassian.com/x/dm2xNQ
        name: Deploy to GCloud
        deployment: production
        image: google/cloud-sdk:latest
        script:
          # Set up credentials
          - echo $GCLOUD_API_KEYFILE > ./gcloud-api-key.json
          - gcloud auth activate-service-account --key-file gcloud-api-key.json
          # Deploy app (and cron if it exists).
          - gcloud config set project $GCLOUD_PROJECT
          - gcloud --quiet --project $GCLOUD_PROJECT app deploy app.yaml
          - if [ -f cron.yaml ]; then gcloud --quiet --project $GCLOUD_PROJECT app deploy cron.yaml; fi

pipelines:
  default:
    - step: *step-test
    - step: *step-build
branches:
  master:
    - step: *step-test
    - step: *step-build
    - step: *step-deploy

Any help would be greatly appreciated, thank you! 

EDIT: I found out that it is my "step-build" that is failing.

EDIT 2: I was able to have it run by combining both my step-test and step-build stages to one step. Are we not allowed to have multiple steps with the same deployment value?

[Kenny MacLeod · Accepted solution]

According to the documentation:

Currently Bitbucket Deployments supports deploying to test, staging, and production type environments and whichever you use they must be listed in this order in each pipeline.

This isn't very well-phrased, but what it means is that only one step can deploy to any given environment, and that the steps must refer to those environments in that order.

Edit:

There's a feature request opened to support multi-step deployments, which is currently in its implementation process:

Support multi-step deployments 

Feel free to vote and watch the suggestion to increase its priority and also receive notifications about any updates.

[Christian Goudreau]

Alright but what if I have more than one step and that I want to use the environment variables for each step?

[Adam James Duffield]

Running into same issue, currently using staging to upload projects to S3. We then wanted a development deployment (I was trying to just re-use staging) and a production environment.

Obviously a development deployment doesn't exist as well as the staging deployment, had to rewrite pipelines to fit to using staging and production in that order. I considered using test too but my workflow would have ended up being staging, test, production which also broke pipelines.

Bottom line, you can only use 1 of each and they have to be in that order, I totally don't agree with this. We should be able to add our own deployments and control the order they can be placed rather than being forced to use test, staging & production.

[Michael Russell]

I just ran into this as well, trying to use the deployment variables for the next step that effects that deploy environment without making the variables global to the repository...

[David Gomez]

So, at least update your bitbucket validator. I validate my file but the pipeline fails. Hasn't sense. 

[raschidjfr]

any update on using deployment variables in various consecutive steps?

[T. Klingenberg]

The one Bitbucket deployment variable I could find that should work in various consecutive steps is `BITBUCKET_DEPLOYMENT_ENVIRONMENT` (and `BITBUCKET_DEPLOYMENT_ENVIRONMENT_UUID`).

No idea why these deployment variables should not be able to use on consecutive deployment steps.

Please share the name of the variables which are missing.

[Michael Russell]

@T. Klingenberg Not talking about the bitbucket provided variables those are injected on every step, we're speaking of variables associated to "Deployments" that are defined by users.

Right now you can create variables for specific deployments, this helps reduce code duplication since you can redefine what the variable is per deployment. However, this breaks down completely and all efficiencies are lost if you need to run another step after a deployment since those deployment defined variables are now gone.

It would be nice to have the ability to do something similar to parallel steps where we could define multiple steps within a deployment phase preserving all the user defined variables for that deployment, or like mentioned above, allow for tagging multiple steps as a deployment type so we can reuse those variables in all associated steps.

[T. Klingenberg]

@Michael Russell Ah, now I understand the problem. You perhaps could persist state between steps in artifacts, this should work if you need multiple steps (e.g. you need different containers for deployments).

[Michael Russell]

Yeah, thought about saving to .env files but we usually have protected vars as well so would rather keep that exposure very limited, especially considering they would now be stored in a downloadable file.

Hoping that the Bitbucket devs can workout a flow for this, it would reduce a lot of redundancies on my end :)

[T. Klingenberg]

@Michael Russell You can always encrypt that with an encryption key for all pipelines if you need this on the same level as "secure" variables. There should be also existing utilities for that, unfortunately I don't have a link at hand.

IMHO it kind of makes sense to only have one step for a deployment. Not suggesting this is it but there is also some value in my eyes to not have the deployment triggering too complicated. Perhaps use a deployment image of your own that ships with all needed utilities for your deployment steps.

[Michael Russell]

I completely agree with:

there is also some value in my eyes to not have the deployment triggering too complicated

Which is why I'd like to see an option of either allowing multiple steps be associated to the same deployment (shares the env vars and the locked down user triggers, which we use in our pro plan) or away to nest steps under a deployment umbrella that has the same effect.

All the solutions at present time can over complicate this. Instead of having to reduce to more complicated methods such as setting up a key vault to encrypt env vars/values to artifacts and then decrypt them upon next steps or building out custom images for everything and embedding all your API keys/tokens/passwords; I see value in multi-step deployments along with single step deployments that are very simple.

I'd rather not make one super complicated step that does it all, but, prefer to allow for multiple, repeatable, testable steps that can build upon each other and be moved around, along with making the yml very human readable and predictable.

Perhaps use a deployment image of your own that ships with all needed utilities for your deployment steps.

We've built out lots of custom images that we use for deployments and we've built out Pipes that run tests for each action on each update we make to them, with proper release cycles; However, even with all this we still can't lock down env variables to specific deployments since lots of these deployments happen across multiple steps for most of our projects. The key for Pipes/custom images to be flexible/secure/testable is to not embed API keys/tokens/etc within those images but allow those images to rely on passing those items into them at execution, which, brings me back to how it'd be nice to have a way to associate multiple steps for deployments along with the simple one step deployment :)

[T. Klingenberg]

I have not yet finished the pipes feature in my local pipelines runner, otherwise it should be possible to run a single step that runs other pipelines. As the environment is shared across all these (IIRC), this should then work. Brings me to the idea to create a pipe for running pipelines.

[Michael Russell]

Yes, it is, however that does not solve for when two steps are needed to complete the deployment process. Thanks for you dedication to this, but, from my end all that's needed is a simple way to have multistep deployment options.

[Myroslav Holyak]

Faced that too doing db migration after app engine deploy via pipe :(

[Mattijs]

I find it a bit weird that I cannot repeat the `deployment` variable in my step.

I had the following definitions:

- step: &build
  name: build
  caches: - node
  script:
    - npm ci
    - npm run aws:config
    - npm run build
    - npm run export

- step: &deploy
  name: deploy 
  caches: - node
  script:
    - npm run deploy

and then in my pipeline I had:

branches:
  master:
    - step:
        <<: *build
         deployment: development
         name: Build for development

    - step:
        <<: *deploy
         deployment: development
         name: Deploy to development




This would trigger the error in :

The deployment environment 'development' in your bitbucket-pipelines.yml file occurs multiple times in the pipeline. Please refer to our documentation for valid environments and their ordering.

But I need the environment in my Build step to access the AWS services for the aws:config, And I need the deployment in my deploy step so that the AWS CDK knows where to deploy too (secret and access keys set in bitbucket deployment environment variables).

So I find this a bit weird, I now have to create a build and a build-deploy step which is kinda defeating creating separate definitions.

[Einar Coutin]

Sorry @Kenny MacLeod but the correct answer should be a link to the Jira ticket:
[BCLOUD-18261] Support multi-step deployments - Create and track feature requests for Atlassian products.

The whole point of a deployment is to have different environment variables.  Nobody wants to have one deployment per step. It renders them either pretty useless or pretty cumbersome.

[Kevin O_Brien]

This was fixed 17/Jul/2023.

I fixed it in my project by changing:

pipelines:
  branches:
    master:
      - step: *init
      - step:
          <<: *build-deploy
          deployment: production
      - step:
          <<: *update-widgets
          deployment: production

To:

pipelines:
  branches:
    master:
      - stage:
          name: Build and deploy live
          deployment: production
          steps:
            - step: *init
            - step:
                <<: *build-deploy
            - step:
                <<: *update-widgets
  

[Igor Ludgero Miura]

Any news on this Atlassian? I need to run 4 differents steps in same deployment, and they are all using different images, but I need the variable that I set on this specific environment.

[T. Klingenberg]

If you set variables, these are only specific to the one script you set them for. I don't understand what to expect otherwise on the Atlassian Bitbucket Cloud Pipelines Plugin honestly. To which reference do you refer here?

[raschidjfr]

@T. Klingenberg , I think @Igor Ludgero Miura refers to deployment variables. Currently you can call deployment vars only from one step, as only one step is allowed to access a deployment in each pipeline.

[T. Klingenberg]

@Igor Ludgero Miura Sure, I was not aware of these specific deployment variables. This should be available, at least this could be seen as very unexpected to have them *not* in further steps if the deployment pipeline allows to have multiple steps so to say. I hope Atlassian support is able to give this some traction. This is perhaps an easy, quick and safe improvement.

[Lin De Lee]

Please work on this. Having to duplicate the same functions for deployments defeats the purpose of using anchors in the yaml file.

[Imran Rais]

Hello Friend. Its funny we all try to do something that is naturally obvious, yet its not according to Atlassian.

[Ash]

@Imran Rais  My thoughts exactly. And they try and justify their reasons with essays of technical jargon rather than a simple "understood, we'll release a (10 line code) update soon".
This is what happens when you have engineers with no UX skills work on a product.

[Julian Rabe]

This can only be a joke, especially because the validator still insists that the yml is valid.

[thisislawatts]

Yes I have just run into the same situation @Julian Rabe, failures in the pipeline but all looks good to the validator. 

[Daniel Ceregatti]

I'm in the same boat here. I need to be able to use 2 different image types across 3 steps for each of test, staging, and production. I can either make my own image, or always install the AWS cli on every deployment, neither of which is ideal.

This seems so arbitrary.

[Daniel Ceregatti]

This was not meant as an answer, but as a reply to the above. Go Atlassian UIs!

[Daniel Ceregatti]

Your own documentation has me doing steps in this way: https://confluence.atlassian.com/bitbucket/deploy-to-amazon-aws-875304040.html

Your instructions for deployments contradict those instructions.

[Sergey K]

We consider the transition to gitlab as a solution to this problem. 

[Jon Berthet]

lol. zinger

[rafaf_tahsin]

signed up and logged in just to upvote your solution @[deleted] 

[Ted Gulesserian]

Ran into same issue here. Having to resort to monolithic pipelines. 

[Jaisa Ram]

i am also facing same issue 

[Musaddiq Ansar]

I'm facing the same issue.  My requirement is as following.
For Production environment deployment, I've bifurcated pipeline into 2 step: First step check and show files to be deployed and then second step actually deploys those files. This is separated into two because I''ve set set step to manuall. So, after confirmation of files-to-be-deployed, I run second step manually.
But this pipeline limitation is now allowing me to do so as deployment command couldn't be repeated as per their error.
So, any update regarding this bitbucket pipeline limitation? Or anyone can help me out to do same with some other way?

[Amir Bengherbi]

I have same issue here, any update regarding that Atlassian?

[Sylwester Gruszka]

Hey Atlassian, how hard can it be to fix this? I'm in the same spot.

Because of this I can't use definitions in more than one step (ones that use vars from deployment env).

[Patrick Wickham]

This is so stupid. How are we supposed to have a manual trigger step to deploy if we can't use the same environment again.

Atlassian over-engineering as usual

[Zeus Camua]

Still havent resolved :( 

Im pretty sure atlassian has the same scenario where they split the steps for one environment. Unless you guys have a monolithic step??? 

[George Mihailoff]

Here is the link to this feature request (or I better say a bug report 😈 )

https://jira.atlassian.com/browse/BCLOUD-18261

[j.lambert]

Why there is an accepted answer to this ???

There is no solution at all and I'm find myself stuck into these issue/bug also.

Where can we vote for the Atlassian team to disable this stupid rule to only allow one step with the deployment variables ??

[George Mihailoff]

https://jira.atlassian.com/browse/BCLOUD-18261

[Mia Paulin]

@Johnnie Kearse Hi, Johnnie, this isn't an answer.  Just checking to make sure everything is ok.  How are you?

[Mia Paulin]

@Johnnie Kearse If you need any help let me know!

[Alexander Högberg]

Same issue here, we have several different dev, stage and prod environments used by the same source. We cannot transition from bamboo to bitbucket pipelines without having the ability to have multiple enviroments in test, staging and production.