nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Bitbucket Dynamic Pipelines initial thoughts

Alex Figliolia

We just finished our first dynamic pipelines Forge app and I have to say I'm very impressed. This couldn't have come at a better time, as we're in a perfect place in our Bamboo to Pipelines transition to take advantage of this.

I do have some thoughts about things that I have come across:

It would be great to have importable JSDoc and/or a good official TypeScript exemplar
We are getting pretty desperate for access to the deployment and environment API endpoints, and will probably end up trying to do something hacky with Forge Remote when it becomes a blocker in the next week or two
It would be nice if there was the ability to put custom configuration nodes into the bitbucket-pipelines.yml file. We tried using labels, but found it far too limiting to satisfy our purposes. We opted instead to have a file checked into the repo with the build setting options.
There seems to be at least one bug in the parsing of the labels. I can take the time to provide a reproduction case if you would like, but when trying to use multi-line strings I would eventually get the error saying that the label key must not be more than 5 characters.
The partial implementation of node-fetch appears to have a bug also. When requesting route`/2.0/repositories/${workspaceId}/${repositoryId}/src/${ref}/.bitbucket/build-rules.yml` we get a permission issue, if I remove the .bitbucket it works fine.
forge lint, even after upgrading forge always returns this error: 2:2 error invalid value 'bitbucket:dynamicPipelinesProvider' in modules valid-module-required - everything proceeds to run fine, I just constantly get that message.

Comments

Edmund Munday

Hi @Alex Figliolia - thanks so much for sharing this detailed feedback! It means more to us than you might think having the community so engaged with these new capabilities, and we're excited that you're seeing this stuff as valuable.

JS-Doc/Typescript: Stay tuned! We're hoping to get `d.ts` files available for all the core entities involved in Dynamic Pipelines in the next week or two. We're focussing on `d.ts` as that will provide auto-complete/validation for both vanilla JS projects AND Typescript projects.
- We're also working on a helper library which will be published as an NPM module which will provide you with helper-functions for pretty much all common API operations + helpers for working with Dynamic Pipelines entities.
Deployment/Environment API support: I'm hoping we'll get all the Pipelines API's fully onboarded for Forge apps in the next 2-3 weeks. I'll speak to the team and make sure Deployments/Environments are in the first release.
Custom config nodes: I'd love to learn more about your specific use-cases here. I'll be honest, I'm a little surprised you're finding the labels limiting as we've used them internally for some VERY comprehensive use-cases (Check this out, you'll find it very interesting: https://www.youtube.com/watch?v=ybzmqZ2ou8U&pp=ygURZHluYW1pYyBwaXBlbGluZXM%3D).
- If you're able to share some details of the specific outcomes you're trying to achieve it would be great!
- One option to scale the capability of labels is to encode more complex configuration as a JSON string and store it as the value on a label. Your Dynamic Pipeline app can then parse that JSON back to useable config.
Bug with labels: If you could share a repro, that would be awesome.
Issue calling `/src` endpoints: This is actually an issue I encountered myself during a hack-day ~2 months ago so I know exactly what you're talking about. This is an issue with the underlying Forge API library which has a bunch of validation to ensure API requests from Forge apps don't break out of the context they're meant to stay within.
- This obviously causes an issue for Bitbucket because of how the `/src` api endpoint works.
- There's a fairly simple workaround for this issue right now which is to URL encode any "/" characters in the file path, which will cause the validation to not get tripped.
  - ```
  const escapedPath = filePath.replace(/\//g, "%2F");
```
- The NPM helper library I mentioned earlier will have this implemented inside it, so you'll just be able to pass the file-path you want to the function and we'll do all the encoding etc.
Issue with forge lint: Have you updated to the latest version of the Forge CLI? Sometimes your local machine can cache an out of date version of it.
- Run the following commands to get a fresh install, it should fix the issue.
- ```
npm uninstall -g @forge/cli
npm i -g @forge/cli@latest
```

Alex Figliolia

@Edmund Munday thank you so much for getting back to me! Here's some more info:

Deployment/Environment API support: For future consideration, I would also like to not have to use both of these endpoints to get the data that I need. It would be great if there was one endpoint that could get the current deployed commit for every environment, and even better to get the tags for each of those commits. I think to do what we want we will need to make a number of requests.
Custom config nodes & bug with labels: Our use cases require quite a lot more flexibility than your example in that video. I'd be happy to go over them in more detail with you, especially now that I have a fully working implementation. However here is a very small example of the bug that I pointed out. This is not a minimal reproduction case, but also gives you an idea of the kind of flexibility we're trying to have in our templates. This bitbucket-piplines.yml gives this error "Label value should not contain more than 5 characters"
```
labels:
scripts:
sharedBuildScript: [
'docker compose -f compose-pipelines.yaml up pipelines-build --exit-code-from pipelines-build --force-recreate',
'docker compose -f compose-pipelines.yaml up asset-builder --exit-code-from asset-builder --force-recreate',
    ]
releaseBuildScript: [
"export DOCKER_BUILDKIT=1",
"docker build ...",
"docker save ...",
"echo \"Prep the artifacts\"",
"mkdir -p dist",
"mv my-app.tar dist/",
"mv compose-*.yaml dist/",
"mv docker/db dist/docker/",
    ]`
```
Issue calling `/src` endpoints: Good to know about the workaround, I figured it was a bug with the babysitter, and I will give your workaround a shot next time I'm working on this later in the week.
Issue with forge lint: I had done an npm update as soon as you released dynamic pipelines. I just now tried uninstall/reinstall and there was no change. Still get the same error, but everything works fine.
Cancel paused manual step: While I have your attention, it would be fantastic if we could cancel or mark invalid a manual step that is paused. Right now the only options are to rerun or run. In our workflows, the manual pauses are for a review process and if the deployment fails that process, everyone just has to know that the deployment is poisonous.
Flexibility for custom deployment variables: It would be great if custom deployment variables allowed for more than only text input. I would be thrilled with the ability to be able to present a select/combo box of allowed variables. Ideally, they could be defined in the pipeline.yml (which now with dynamic pipelines would be so powerful). If that's not feasible, letting the user choose from a dropdown of refs (specifically tags) would also be an improvement. We're still really struggling to reproduce the flexibility that we had with the split between builds and deployments with Bamboo, and we're trying different strategies to "hack" them into place with Pipelines.

Alex Figliolia

@Edmund Munday When I reran the forge install it created a ~/.bash_profile file which caused my ~/.profile to not be run.

It was easy for me to fix once I realized what happened, but it was confusing and may cause issues for people.

Edmund Munday

Hey @Alex Figliolia -

Deployments/Environments: Thanks for the extra info, we'll have a think about ways it would make sense to incorporate it. Just to set expectations, the initial support for these endpoints in Forge will be the same API's as are available today as that allows us to move quickly.
- Other API's we've got do support the concept of "expands" though where you can drill-down into related data. This is something we might be able to consider in the future.
Issues with labels: So, I dug into this with the team yesterday and identified the cause of the issue. Without going super deep into detail, the way the system parses lists of items in .yaml means that it was very easy to blow out the previous "per value" 128 character limit that labels enforced.
- The way we are enforcing limits on labels today is quite arbitrary and not super fit for purpose (to be totally honest, "labels" as a feature are still technically undocumented and not "officially" released haha).
  - Previously there was a bunch of limits like "max tree depth", "max items per level of the tree", "max characters per value", etc.
- We're going to drastically simplify the limits and validation done on labels to make them a lot more flexible. Rather than enforcing a bunch of different limits for each dimension of the key/value tree, we're just going to calculate the total size of the labels object in KB and set an upper boundary.
  - The limit we are currently planning to implement would roughly equate to around 200 lines of moderately complex JSON when parsed.
- If a use-case needs more than this, it would be a pretty strong signal that the overall architecture of the implementation is probably sub-optimal, and you should consider alternatives like storing larger blocks of pre-baked config directly in the code of the Dynamic Pipeline app and then referencing it with a label, or using Forge Storage + some Forge UI to allow the app to store more complex dynamic configuration in the database that Forge provides natively.
Issue with forge lint: Are you using an IntelliJ based IDE by any chance? If you're using something like NVM your IDE might be pointing at a different Node environment vs your stand-alone terminal, which means that when you do the update from your terminal it would be updated in one environment, but not in the other.
- If it's not immediately obvious from my tone... I spent the last 3 days being confounded by the exact same issue 😅
Cancel Paused Step: Yeah that makes a lot of sense! I can't promise anything super specific, but will definitely add this to the list.
Custom Deployment Variables: That would be a pretty useful capability actually! 😉
- Following on from this though, another option we could look at in the future is powering this from something like a Forge function where you specify a Forge function in the .yaml file to call, that function can then generate a list of options at invocation time (which means it would be totally dynamic, could pull data in from other systems, etc etc), and return that list of values as a simple list of strings.
- Would be curious to get your thoughts on something like that.

Alex Figliolia

@Edmund Munday I forgot about the allowed-values since it had limited use when it was static. I played with it for a while yesterday and it does a great job of handling my immediate needs (along with your hack for encoding the / in the API URL).

I'm sure you're getting tired of my requests, but here is what I would love to see for the allowed-values (these are mostly QOL things):

Allow both a label and a value instead of just a value.
Instead of requiring a default, have an option for a required variable that just won't submit the form unless it's chosen. Right now I have to make a "Choose One" value, set it as default, and then have the step script fail if they didn't set it.
I don't know if this is feasible, but it would be nice to get the values that were selected for variables passed into the request object of the dynamicPipelinesProvider. That way for pipelines that have a 'manual' trigger, the pipeline could be customized.

Edmund Munday

Hi all - just a quick note to let everyone know that we've now added support for Deployment, Environment, Environment Variable, and Repo variable API's for Forge apps.

Details are here: https://community.atlassian.com/t5/Forge-for-Bitbucket-Cloud/Additional-support-for-Pipelines-API-s-from-Forge-Apps-for/ba-p/2696769

Jan Hoppel

Hi @Edmund Munday , a bit late to the party but do you have any update regarding types for dynamic pipelines? Not having to debug the requests and creating our own types would help tremendously.

Best,

Jan

Joseph Terlecki

Hi @Edmund Munday

Has there been any updates regarding d.ts type declartions or the npm package?

We have started needing the advanced functionality of dynamic pipelines. While the experience has been decent having propper intellisense and autocomplete would be alot easier to manage and develop

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket

bitbucket-pipelines

forge