Bitbucket Dynamic Pipelines initial thoughts

[Alex Figliolia]

We just finished our first dynamic pipelines Forge app and I have to say I'm very impressed. This couldn't have come at a better time, as we're in a perfect place in our Bamboo to Pipelines transition to take advantage of this.

I do have some thoughts about things that I have come across:

1. It would be great to have importable JSDoc and/or a good official TypeScript exemplar
2. We are getting pretty desperate for access to the deployment and environment API endpoints, and will probably end up trying to do something hacky with Forge Remote when it becomes a blocker in the next week or two
3. It would be nice if there was the ability to put custom configuration nodes into the bitbucket-pipelines.yml file. We tried using labels, but found it far too limiting to satisfy our purposes. We opted instead to have a file checked into the repo with the build setting options.
4. There seems to be at least one bug in the parsing of the labels. I can take the time to provide a reproduction case if you would like, but when trying to use multi-line strings I would eventually get the error saying that the label key must not be more than 5 characters.
5. The partial implementation of node-fetch appears to have a bug also. When requesting route`/2.0/repositories/${workspaceId}/${repositoryId}/src/${ref}/.bitbucket/build-rules.yml` we get a permission issue, if I remove the .bitbucket it works fine.
6. forge lint, even after upgrading forge always returns this error: 2:2 error invalid value 'bitbucket:dynamicPipelinesProvider' in modules valid-module-required - everything proceeds to run fine, I just constantly get that message.

[Edmund Munday]

Hi @Alex Figliolia - thanks so much for sharing this detailed feedback! It means more to us than you might think having the community so engaged with these new capabilities, and we're excited that you're seeing this stuff as valuable.

• JS-Doc/Typescript: Stay tuned! We're hoping to get `d.ts` files available for all the core entities involved in Dynamic Pipelines in the next week or two. We're focussing on `d.ts` as that will provide auto-complete/validation for both vanilla JS projects AND Typescript projects.
  • We're also working on a helper library which will be published as an NPM module which will provide you with helper-functions for pretty much all common API operations + helpers for working with Dynamic Pipelines entities.
• Deployment/Environment API support: I'm hoping we'll get all the Pipelines API's fully onboarded for Forge apps in the next 2-3 weeks. I'll speak to the team and make sure Deployments/Environments are in the first release.
• Custom config nodes: I'd love to learn more about your specific use-cases here. I'll be honest, I'm a little surprised you're finding the labels limiting as we've used them internally for some VERY comprehensive use-cases (Check this out, you'll find it very interesting: https://www.youtube.com/watch?v=ybzmqZ2ou8U&pp=ygURZHluYW1pYyBwaXBlbGluZXM%3D).
  • If you're able to share some details of the specific outcomes you're trying to achieve it would be great! 
  • One option to scale the capability of labels is to encode more complex configuration as a JSON string and store it as the value on a label. Your Dynamic Pipeline app can then parse that JSON back to useable config.
• Bug with labels: If you could share a repro, that would be awesome.
• Issue calling `/src` endpoints: This is actually an issue I encountered myself during a hack-day ~2 months ago so I know exactly what you're talking about. This is an issue with the underlying Forge API library which has a bunch of validation to ensure API requests from Forge apps don't break out of the context they're meant to stay within.
  • This obviously causes an issue for Bitbucket because of how the `/src` api endpoint works.
  • There's a fairly simple workaround for this issue right now which is to URL encode any "/" characters in the file path, which will cause the validation to not get tripped.
    

    • const escapedPath = filePath.replace(/\//g, "%2F");

  • The NPM helper library I mentioned earlier will have this implemented inside it, so you'll just be able to pass the file-path you want to the function and we'll do all the encoding etc.
• Issue with forge lint: Have you updated to the latest version of the Forge CLI? Sometimes your local machine can cache an out of date version of it. 
  • Run the following commands to get a fresh install, it should fix the issue.

  • npm uninstall -g @forge/cli
    npm i -g @forge/cli@latest

[Alex Figliolia]

@Edmund Munday thank you so much for getting back to me! Here's some more info:

• Deployment/Environment API support: For future consideration, I would also like to not have to use both of these endpoints to get the data that I need. It would be great if there was one endpoint that could get the current deployed commit for every environment, and even better to get the tags for each of those commits. I think to do what we want we will need to make a number of requests.
• Custom config nodes & bug with labels: Our use cases require quite a lot more flexibility than your example in that video. I'd be happy to go over them in more detail with you, especially now that I have a fully working implementation. However here is a very small example of the bug that I pointed out. This is not a minimal reproduction case, but also gives you an idea of the kind of flexibility we're trying to have in our templates. This bitbucket-piplines.yml gives this error "Label value should not contain more than 5 characters"
  
  
  

  labels:
    scripts:
      sharedBuildScript: [
        'docker compose -f compose-pipelines.yaml up pipelines-build --exit-code-from pipelines-build --force-recreate',
        'docker compose -f compose-pipelines.yaml up asset-builder --exit-code-from asset-builder --force-recreate',
      ]
      releaseBuildScript: [
        "export DOCKER_BUILDKIT=1",
        "docker build ...",
        "docker save ...",
        "echo \"Prep the artifacts\"",
        "mkdir -p dist",
        "mv my-app.tar dist/",
        "mv compose-*.yaml dist/",
        "mv docker/db dist/docker/",
      ]`

• Issue calling `/src` endpoints: Good to know about the workaround, I figured it was a bug with the babysitter, and I will give your workaround a shot next time I'm working on this later in the week.
• Issue with forge lint: I had done an npm update as soon as you released dynamic pipelines. I just now tried uninstall/reinstall and there was no change. Still get the same error, but everything works fine.
  
• Cancel paused manual step: While I have your attention, it would be fantastic if we could cancel or mark invalid a manual step that is paused. Right now the only options are to rerun or run. In our workflows, the manual pauses are for a review process and if the deployment fails that process, everyone just has to know that the deployment is poisonous.
  
• Flexibility for custom deployment variables: It would be great if custom deployment variables allowed for more than only text input. I would be thrilled with the ability to be able to present a select/combo box of allowed variables. Ideally, they could be defined in the pipeline.yml (which now with dynamic pipelines would be so powerful). If that's not feasible, letting the user choose from a dropdown of refs (specifically tags) would also be an improvement. We're still really struggling to reproduce the flexibility that we had with the split between builds and deployments with Bamboo, and we're trying different strategies to "hack" them into place with Pipelines.

[Alex Figliolia]

@Edmund Munday When I reran the forge install it created a ~/.bash_profile file which caused my ~/.profile to not be run. 

It was easy for me to fix once I realized what happened, but it was confusing and may cause issues for people.

[Edmund Munday]

Hey @Alex Figliolia -

• Deployments/Environments: Thanks for the extra info, we'll have a think about ways it would make sense to incorporate it. Just to set expectations, the initial support for these endpoints in Forge will be the same API's as are available today as that allows us to move quickly.
  • Other API's we've got do support the concept of "expands" though where you can drill-down into related data. This is something we might be able to consider in the future.
• Issues with labels: So, I dug into this with the team yesterday and identified the cause of the issue. Without going super deep into detail, the way the system parses lists of items in .yaml means that it was very easy to blow out the previous "per value" 128 character limit that labels enforced.
  • The way we are enforcing limits on labels today is quite arbitrary and not super fit for purpose (to be totally honest, "labels" as a feature are still technically undocumented and not "officially" released haha).
    • Previously there was a bunch of limits like "max tree depth", "max items per level of the tree", "max characters per value", etc.
  • We're going to drastically simplify the limits and validation done on labels to make them a lot more flexible. Rather than enforcing a bunch of different limits for each dimension of the key/value tree, we're just going to calculate the total size of the labels object in KB and set an upper boundary.
    • The limit we are currently planning to implement would roughly equate to around 200 lines of moderately complex JSON when parsed.
  • If a use-case needs more than this, it would be a pretty strong signal that the overall architecture of the implementation is probably sub-optimal, and you should consider alternatives like storing larger blocks of pre-baked config directly in the code of the Dynamic Pipeline app and then referencing it with a label, or using Forge Storage + some Forge UI to allow the app to store more complex dynamic configuration in the database that Forge provides natively.
• Issue with forge lint: Are you using an IntelliJ based IDE by any chance? If you're using something like NVM your IDE might be pointing at a different Node environment vs your stand-alone terminal, which means that when you do the update from your terminal it would be updated in one environment, but not in the other.
  • If it's not immediately obvious from my tone... I spent the last 3 days being confounded by the exact same issue 😅
• Cancel Paused Step: Yeah that makes a lot of sense! I can't promise anything super specific, but will definitely add this to the list.
• Custom Deployment Variables: That would be a pretty useful capability actually! 😉
  • Following on from this though, another option we could look at in the future is powering this from something like a Forge function where you specify a Forge function in the .yaml file to call, that function can then generate a list of options at invocation time (which means it would be totally dynamic, could pull data in from other systems, etc etc), and return that list of values as a simple list of strings.
  • Would be curious to get your thoughts on something like that.

[Alex Figliolia]

@Edmund Munday 

• Deployments/Environments: I fully understand, the change to the API I was suggesting was just a "nice to have". Needing to be able to call these endpoints, as they are currently implemented, is what is our blocker.
• Issues with labels: I think a limit of about 200 lines would work perfectly. Here's an example from one of our more complex applications that we've moved over to dynamic pipelines; the original yml file was 526 lines and 22.5KB. After the move, the build rules yml file was 66 lines and less than 3k.
  While we could certainly reduce the number of lines more, we wanted to leave a lot of flexibility in the build processes. We have a lot of variety in our repos, build, and deployment needs.
  I wasn't aware of the Forge storage API. I'll have to meditate on that for a while to see if it will be a good solution for any of our other issues. I can certainly see it helping with some caching, but I don't think we'll go that way for build configurations. Our current solution of moving the configuration to be a part of the repo is working well for the interim.
  
• Issue with forge lint: Yes I use IJ, and yes I use nvm, and yes I fight with it a lot. Not as bad as using nvm with XCode though where the only solution I've found is to hard code your path to node into the project files and make sure you never commit them.
  Yesterday Forge told me it was out of date and to reinstall to version 9.0.0. After upgrading I had to update the manifest.yml to set the runtime to get it to run, but after that, I have had no problems and the previous error went away.
• Custom Deployment Variables: I think that this is more of an architectural question specific to Bitbucket. I don't see much of a practical difference in having a function just for generating the options, or putting the options in the pipelines yml and being able to replace it with the dynamic pipelines. It depends on the timing of where the UI needs the data.
  The values for the combo box are tightly coupled to the build process, so putting it in the dynamic pipelines will probably save on some redundant API calls. But, if the UI needs to make another call after you choose the custom pipeline to get the variable parameter data, then you may as well make it a separate function.
  Another exciting possibility would be the ability to supply both the value and the label for the items in the options, for example for a production deployment custom pipeline it could have "v3.0.5 (currently deployed to staging)".
  I'd be very happy with this either way, and my suggestion would be to choose the implementation that you can get built the quickest so that I can start using it!
  
• Issue with Dynamic Pipelines Repository Settings page: On the repository settings page, the select box for "Repository dynamic pipelines provider" shows 3 copies of my app. I'm almost certain that these are the 3 environments (dev/stage/prod), but they're not labeled. The only way I can know which one I am choosing is to guess and then check logs.
• Issue on the Pipeline log page: Another new issue. I've noticed in the last couple of days that the build duration on the execution log doesn't update while I'm watching the page during a build anymore. I have to reload the page to get the build time to be accurate.

[Edmund Munday]

Thanks @Alex Figliolia 

Issues with labels: Thanks for the extra details, this is exactly what we needed to validate our thinking. The current plan is for the limit to be 10KB once fully parsed to JSON.

Custom Deployment Variables: Just for my context (sorry if I’m missing something!), does the changes we made to give you a static dropdown of options for custom variables talked about in this blog cover what you need in the immediate term?

Issue with Dynamic Pipelines Repository Settings page: Thanks for raising this, I’ll take a look at this and we’ll add something in that makes sense.

Issue on the Pipeline log page: This is a new one to me, we’ll take a look and see what’s going on.

[Alex Figliolia]

@Edmund Munday I forgot about the allowed-values since it had limited use when it was static. I played with it for a while yesterday and it does a great job of handling my immediate needs (along with your hack for encoding the / in the API URL). 

I'm sure you're getting tired of my requests, but here is what I would love to see for the allowed-values (these are mostly QOL things):

1. Allow both a label and a value instead of just a value.
2. Instead of requiring a default, have an option for a required variable that just won't submit the form unless it's chosen. Right now I have to make a "Choose One" value, set it as default, and then have the step script fail if they didn't set it.
3. I don't know if this is feasible, but it would be nice to get the values that were selected for variables passed into the request object of the dynamicPipelinesProvider. That way for pipelines that have a 'manual' trigger, the pipeline could be customized.

[Edmund Munday]

Hi all - just a quick note to let everyone know that we've now added support for Deployment, Environment, Environment Variable, and Repo variable API's for Forge apps.

Details are here: https://community.atlassian.com/t5/Forge-for-Bitbucket-Cloud/Additional-support-for-Pipelines-API-s-from-Forge-Apps-for/ba-p/2696769

[Jan Hoppel]

Hi @Edmund Munday , a bit late to the party but do you have any update regarding types for dynamic pipelines? Not having to debug the requests and creating our own types would help tremendously. 

Best,

Jan