🤖 What we learned building AI into a Forge Marketplace app

[#1]

When we first integrated the Forge LLMs API into Jira Workflow Toolbox, it was still in Early Access — and we were among the first Marketplace partners to do so. Last week, Atlassian made the Forge LLMs API generally available, and we were glad to see JWT explicitly named as one of the early adopters in their announcement. That felt like a good moment to share what the journey actually looked like: the product problem we started with, the engineering decisions we made along the way, and the lessons that only became clear once real customers started using it.

The product problem came first

We started with a specific problem in Jira Workflow Toolbox: JWT Expressions are powerful, but they come with a learning curve. Administrators often need to move between documentation, examples, trial and error, and syntax validation before an expression works.

We had already added autocomplete, built-in documentation, copy-and-paste examples, and field code injection. Those features helped, but creating a complex expression still required users to understand JWT syntax.

That made expression assistance a practical use case for AI. Users could describe a requirement in natural language, generate an expression, and ask for an explanation of existing expressions without leaving the editor.

Why we chose Forge LLM

We joined the Forge LLM Early Access Program before the public announcement. The program gave us time to test the platform, refine the user experience, and share feedback with Atlassian while the module was still changing.

We chose Forge LLM instead of connecting JWT to an external provider. Jira Workflow Toolbox already uses Forge for application logic, feature control, and tenant-aware behaviour. Using the same platform meant we could keep the AI request in the existing Forge resolver flow rather than add another service, authentication layer, or operational boundary.

The choice also made the architecture easier to explain to enterprise customers. Atlassian hosts the model, and the integration remains within the platform boundaries customers already use for Forge applications. That helps with questions about data handling, compliance, and security, and supports our longer-term goal of achieving the Runs on Atlassian badge.

What the Expression Generator does

The AI-powered Expression Generator is built into the Jira Workflow Toolbox Expression Editor. An administrator can describe a requirement such as:

Only allow this transition if the priority is High and an assignee has been set.

Only allow this transition if there is at least one attachment.

The assistant turns the request into a JWT expression that can be inserted into the editor.

Explanation has become an equally important use case. Administrators often inherit workflows containing expressions written years earlier by someone who is no longer with the organisation. The assistant can describe what an existing expression does in plain language, answer follow-up questions, and help users refine a generated result.

From early access to Marketplace

Our work began before Forge LLM entered Preview. Early product discussions identified expression assistance as a focused use case, and the Early Access Program let us build and test a prototype while the platform evolved.

Turning that prototype into a Marketplace-ready feature required more than the initial development team. Product management, prototype developers, the JWT Cloud team, support, and operations all contributed. About twenty people were involved at different stages.

How the request works

At a high level, the request flow is:

The model connection was straightforward. The harder work was making the output reliable enough for production.

• Jira Workflow Toolbox provides the editor, user experience, and domain-specific context.

• The request is enriched with JWT syntax, available functions, notation rules, field mappings, and customer-specific context.

• Forge provides the communication framework.

• Atlassian hosts the model.

• Jira Workflow Toolbox validates the resulting expression and returns it to the editor.

Context matters more than clever prompting

Language models can produce expressions that look plausible without being valid JWT or appropriate for a particular Jira configuration. We therefore built a structured instruction layer instead of relying on a single prompt.

The system prompt includes:

• General guidance for the assistant.

• JWT syntax, notation rules, available functions, and common mistakes.

• Customer-specific field information, including field IDs and data types.

This context limits the model's assumptions and gives it the information needed to produce expressions that match the customer's configuration. In our tests, product-specific context had a larger effect on quality than simply switching to a larger or more expensive model.

Validation is part of the feature

Users need more than a plausible expression. They need an expression that passes the product's rules.

The Expression Generator therefore has two modes:

1. Fast returns the model's first response.

2. Validated Results sends the generated expression through Jira Workflow Toolbox's existing syntax validator.

If validation fails, the assistant receives the errors and its previous answer, then tries again. The process runs for up to three rounds before the result is shown. This combines probabilistic generation with deterministic validation and gives users a clearer basis for deciding whether to use the expression.

Engineering lessons from real usage

The main engineering work was defining the boundaries of the assistant. We had to specify what a valid answer looks like, which expression mode to use, which rules cannot be relaxed, and when the assistant should ask for clarification or avoid assumptions.

Token usage was another early surprise. Before inviting preview customers, we introduced usage tiers to control costs. The lowest tier was intended to support several prompts, but the first customer exceeded it with one request. The request itself was not unusually large; the customer's Jira instance contained many custom fields, and the field information was included in the prompt.

Our test environments did not reflect the size of some real customer configurations. We had to account for that difference before expanding the preview.

Security and data handling

The assistant receives only the information needed to generate or explain an expression:

• The user's prompt.

• Field names and IDs.

• Field types.

• The expression context.

Field mappings are necessary because users refer to fields by name while JWT expressions use the underlying IDs and data types. The model is hosted by Atlassian, and communication runs through Forge. Keeping the interaction within the Atlassian platform gives the feature the same trust foundation as the rest of the Forge application.

What we learned

Shipping an AI feature inside a production product required work across the model, the application, and the surrounding user experience. The main reliability improvements came from:

• Guardrails.

• Deterministic validation.

• JWT-specific knowledge.

• Clear limits on what the assistant should do.

• Integration with the existing expression editor.

The model is only one part of the system. The surrounding product determines what context the model receives, how its output is checked, and how much control the user retains.

What we would do again

If we started the project again, we would keep the same general architecture and apply these lessons earlier:

• Start with a concrete product problem.

• Integrate AI into an existing workflow.

• Invest in domain knowledge and context.

• Validate outputs whenever a deterministic validator is available.

• Test with realistic customer configurations early.

• Define what the assistant must not do.

What's next

The current release is an early preview. We are using customer feedback to measure which prompts work, where users need clarification, which scenarios remain difficult, and how much time the assistant saves.

One early pattern is clear: customers use the assistant to explain existing configurations as well as generate new expressions. In large Jira instances with years of accumulated workflow logic, that may be as valuable as generation.

We plan to improve result quality, evaluate new models, extend AI capabilities to other Jira Workflow Toolbox features, and work toward General Availability later this year.

Try it yourself

The AI-powered Expression Generator is available as an early preview behind a feature flag.

If you already use Jira Workflow Toolbox Cloud, you can request preview access here: https://go.decadis.net/jwtc/ai-assistant

And if you're not using it yet, make sure, to check out our Marketplace listing here: https://go.decadis.net/jwtc/marketplace

We are especially interested in feedback about:

• Prompts that produce useful results.

• Scenarios where the assistant struggles.

• Clarifications users need.

• Changes that would make the feature more useful in daily work.

That feedback will guide the next version of the Expression Generator.

[Mia Tamm _Simpleasyty_]

Really interesting post @Thorsten Letschert _Decadis AG_.

What I liked most is that the AI is helping inside the existing workflow rather than trying to replace it. Keeping validation and user control in the process makes the feature feel much more useful and trustworthy.

Also appreciated the engineering lessons at the end — those details are often the most valuable part of posts like this.

Thanks for sharing!