Introducing Dynamic Pipelines: A new standard in CI/CD flexibility

[#1]

Hi all! It's been a very busy day in the land of Forge for Bitbucket Cloud.

The first major announcement we have for today is the GA release of an entire new area of capability inside Bitbucket Cloud leveraging Forge called Dynamic Pipelines!

Dynamic Pipelines allow you to utilise Forge functions to augment and modify CI/CD workflows at runtime, implementing sophisticated dynamic logic that will enable unparalleled levels of CI/CD control.

What's more, Dynamic Pipelines have been built to support a powerful hierarchical orchestration pattern, allowing runtime Pipeline modification to be done both at the repository level (for individual teams and repos), AND at the workspace-level for "whole-of-organisation" CI/CD orchestration.

Check out all the details and information on how to get started over at our official release blog.

[Michael Rüegg _Mibex Software_]

Thanks @Edmund Munday . Excited to try this out.

Question: are there any plans to support Connect on Forge for Bitbucket Cloud, so that we can use these nice new features without having to fully migrate existing Bitbucket apps to Forge?

[Edmund Munday]

Hi @Michael Rüegg _Mibex Software_

As of today, it's unfortunately not something we're investing heavily in for Bitbucket cloud, for two primary reasons.

1. Bitbucket Cloud's traditional Connect implementation is not 100% standardised with the wider Atlassian Connect ecosystem (unlike our Forge implementation, which is 100% standard), so the process for other products is not 100% reusable for Bitbucket.
2. We are, at the moment, focussing all our surplus effort onto trying to accelerate paid marketplace support for Bitbucket cloud. We've taken the view right now that customers and partners would benefit more from the ability to do PVA Marketplace apps vs "Connect on Forge" - but I'd love to hear your opinion on that.

All that being said, we do 100% support Forge Remote, which (unofficially) is a really powerful tool to support an incremental Connect => Forge migration.

[Michael Rüegg _Mibex Software_]

@Edmund MundayThanks, getting PvA is absolutely key, so I'm 100% fine with your prioritization.

Besides PvA, another (limiting) key factor for enterprise adoption of Bitbucket Cloud apps is the fact that rate limits are still based on the user that installed the app and not the current user. This results in rate limit exceptions for larger workspaces. The current workaround is to ask customers to install the apps with different users and not with the same user to somewhat deal with this limitation. I don't think this is a good user experience, and it is not really fixing the issue. Do you have that on the agenda as well? Is there a public issue for this we can watch, or do you want me to create one?

[Edmund Munday]

Hey @Michael Rüegg _Mibex Software_ - another excellent question!

There's a bunch of important information to share regarding rate limits, so I'll put it all here.

• Firstly, the way API rate limits are calculated for Forge apps is totally different to Connect apps.
  • Each Forge app is an entirely seperate "identity" and has it's own API rate limit allowance for each seperate Workspace.
  • Basically what this means is you no longer have any issues with API rate limit "leakage", where usage from apps could, depending on installation context, affect the ability of OTHER apps to access the API.
  • This change has been in affect since we went into open-beta with Forge last year.
• Unfortunately this change is not going to be ported to Connect as it would require close to a ground-up rebuild of the Connect app identity model, and we're focussing our extensibility efforts on Forge.
  • While I understand this will be disappointing, I would like to hope that the progress we've made and the direction we've pushed in over the last 12 months reassures the community that we listening very intently.
  • Our #1 objective right now as far as our partner community is concerned is to carve a path as directly as possible towards a fully integrated and fully Atlassian standard PvA marketplace implementation inside BBC.
    • The unfortunate reality is that the way we get this done ASAP, is by focussing exclusively on Forge as our foundational extensibility platform for Bitbucket Cloud.
• We are also very aware that without workable API rate limits, there's a very real glass-ceiling on the value apps can provide in BBC, no matter which framework or platform they're running on.
  
  • The current API rate limit allowances are, to be blunt, totally not fit-for-purpose. They made sense in a world where Bitbucket Cloud was primarily focussed on small/medium sized teams of 20-50; but as we acquire more and more large customers with the migration off Server/DC, it's obvious things need to change.
    
    • Fun fact - Bitbucket Cloud's largest customer now (Other than Atlassian itself) has over 6000 seats.
  • We are planning to kick off work in the next few months on a project to rework our API rate limits, whilst still ensuring we can maintain the same quality of service and SLO's we have today.
  • The high-level details of how this will look are something along the lines of a tiered API rate limit, possibly pinned to the number of users active within a given Bitbucket Cloud workspace. So basically, if you're a bigger workspace with more users, the rate-limits you have access to will be higher.
    • I want to be really clear - none of this is confirmed, it is all 100% liable to change as we get into the details more.
  • The exact numbers are not yet confirmed, but a rough idea of the ratios would be something in the area of:
    • <100 users - No change
    • 100 => 250 users - 2x current
    • 250 => 1000 users - 5x current
    • >1000 users - 10x current.
• When you combine these higher limits with the first point around better app-level API consumption isolation, we're confident this will result in a dramatic improvement of all the issues being experienced today around API rate limits.
• One last note: these changes to rate limits will most likely NOT apply to user-level App Passwords, they will only be for API access mechanisms that are designed to operate across an entire workspace (like Apps & Rep/Project/Workspace Access Tokens), where the API call counts would be expected to scale fairly linearly with workspace size.
  • As far as we are concerned, there's no real need for an individual user to have an API rate limit of >1000 an hour, unless that user's App Password is actually being used for some kind of "service account" type workload, in which case something like a Forge app or a Repo/Project/Workspace Access Token is a more appropriate option.

[Michael Rüegg _Mibex Software_]

Thanks @Edmund Munday , I'm glad to hear the focus on a fully integrated and fully Atlassian standard PvA marketplace implementation inside BBC. Given this, we will start migrating existing BBC Connect apps to Forge as soon as possible to overcome the current rate limit issues. Seeing PvA is on the horizon, it helps us a lot to do this prioritization.