Disclosure: I work at LeanZero, which built this benchmark, and the model results below come from our own runs.
In August a member here wrote up their experience of building a WIP limit app for Jira with Rovo Studio, and I've kept thinking about it since. The first version deployed and looked finished, yet in their own list of what wasn't working the top line reads: "The validator itself is not preventing the transition." They tracked it down to two quiet problems: a search request the API rejects, which the generated code answered by letting the transition through, and a count the newer search endpoint doesn't return. With their own Forge and REST knowledge they got it working in the end. You can follow the whole story in Vibe Coding my WIP Limit Enforcement Forge App with Rovo Studio.
That's the part that matters for those of us who approve what goes on a Jira site. The app installed cleanly, and nothing on the surface told you it wasn't doing its job until someone actually tried to break the limit; working out why then took real platform knowledge. So when our team built a benchmark that asks AI models to build a full Forge app, the leaderboard wasn't what I cared about most. I wanted to know how you'd judge an app like that without taking anyone's word for it.
The app every model had to build
The brief was the same for every model, and I think you'll recognise the kind of request: an app for the agile coach on a Jira team that keeps a running record of work added to a sprint after it began, with the person who added it and the story points it brought along. Coaches see it on a dashboard, in a dialog opened from the sprint's own menu, and through Rovo. Atlassian's hosted AI service, Forge LLM, is expected to explain in plain words why the sprint grew. Ten Forge module types are involved, plus the Realtime API, so it's the size of something a team would ask for.
The models worked with no internet connection, which I think is the fairest way to see what they really know. They could use Forge's own linter, the manifest schema, Jira's API specification and a mock Jira site to try things against, but they couldn't open Atlassian's developer site or look for a tutorial. Whatever they knew about Forge had to come from what they'd already learned or from the type definitions in front of them.
How the apps were judged without anyone reading the code
No one sat down and reviewed the code; the point was to see how much you can learn without doing that. Apart from Forge's linter and a set of rules checked against the manifest and source code, the grade comes from running the app: it executes inside an emulator built on Atlassian's own Forge runtime, against three freshly seeded mock Jira sites. Every request it sends to Jira passes through a proxy that writes it down, and the scorer works from that record. The screens are opened in a real browser, in light and dark themes, screenshotted and recorded on video.
The mock sites are set up to be awkward on purpose, because production sites usually are. Most of what gets checked is something I'd want any app on my own site to handle, whoever or whatever wrote it:
- Can someone see an issue through the app that they couldn't open in Jira? An app that fetches data under its own identity, instead of the identity of the person looking, can leak restricted issues into a dashboard without anyone noticing.
- When somebody double-clicks a button that writes a comment, does exactly one comment land on the issue?
- When Jira rate-limits the app and tells it to wait 30 seconds, does it wait and carry on?
- Is it built on today's platform? Jira's old search endpoint has been removed, and the old dashboard gadget module is deprecated with removal set for 17 May 2027, now that the dashboard widget module has taken its place.
- Does every screen follow the theme, so that nothing turns into a glaring white panel for your dark-mode users?
- How does it handle the AI service misbehaving? The mock service replies once properly, once with invented figures and a restricted issue slipped in, and then with a refusal, a garbled reply and an outright error. A careful app shows the good reply, cleans the invented details out of the second, and shows a calm error message for the last three.
The most serious failures, such as a restricted issue reaching the wrong person or a duplicate comment on someone's issue, cut the whole score by a multiplier.
How the eleven apps did
By early Monday, 5 October, eleven models had a result on the Forge board. GPT-6.1 Sol, Claude Opus 5.5 and GPT-6.1 Sol Pro each scored above 0.96, and nothing else reached 0.80. We publish the per-model results, with every run's individual checks, on our own site.
The run I'd show any admin is Claude Sonnet 5.5's, at 0.5508. Its data work was every bit as solid as the leaders': it processed each change event correctly, recovered the events the mock site deliberately dropped, and never showed a restricted issue to the wrong person. Where it fell down was the part people actually click. Double-clicking its post button, straight after a first post that worked, produced no comment at all, its explain button didn't reach the AI service once in five tries, and on one of the mock sites its close button didn't close the dialog. In fairness, our brief never said whether the change you'd just clicked in the sprint dialog should stay selected after a post, so part of the double-click miss may be on us, and we haven't dug into its code to see why any of the three failed. Still, if you'd only checked the data, you'd have signed it off.
The app in fourth place earned 0.9762 and was capped by a single screen, finishing at 0.7978. Its "this sprint has not started" screen shows a single line of text and a button, and our theme check read a screenshot that's nearly all background as a sign nothing had rendered. Having looked at that screenshot, I'd call it a mistake in our scorer, not in the app. Lower down, one app scored 0.0292 because nothing in it ever ran: its code had an error that stopped both of its functions from loading, and its configuration pointed at screen files that didn't exist. On the reassuring side, the restricted-issue check held for the strongest apps, and, as the brief asked, GPT-6.1 Sol's sprint dialog told the viewer that 2 changes were hidden by issue permissions.
Each model has only one run on the board so far, and new runs are still landing, so I'd read the ranking loosely; if you compare two close scores, they could easily swap places on the next run.
What I'd ask for when someone hands you an AI-built app
"It passes lint" and "it deployed" tell you very little, and the WIP limit app above deployed fine. In most cases the failures that matter surface once a real person starts clicking, so I'd test it the way your users will meet it. Use a test account with restricted permissions and look at what it shows, and double-click every button that writes something. Switch to dark mode, then ask which Forge modules it uses and whether any of them are on Atlassian's deprecation list. If the person who built it can't tell you how it was tested on a running site, that's usually your answer.
The same lesson applies to AI agents
We also train our own Atlassian model, and what it taught us applies to agents in Jira and Confluence as much as to apps. The version we trained last week was chosen after our own checks and approved for release, and then its final release checks caught habits you wouldn't want anywhere near a service desk. Given ordinary admin questions, it now and then began its answer by greeting a forum username it had made up. When we asked it about a particular company, it filled the gaps with fiction: ticket keys assembled out of that company's initials, a site address with an invented cloud ID, and incidents that never took place.
We didn't publish that version. Its predecessor did both of those things too, and had picked up the greeting from forum threads in its training data, so we started again from the original base model. That predecessor is a later round of the model I wrote about here in September. If you're running it, or evaluating any AI agent on your own Jira or Confluence, I'd ask it about your own organisation and check that every ticket key and every person it names really exists.
That restart was published earlier today, and its own last check caught something quite different. Asked for 35 Forge apps, with three tries at each, it averaged only 9.7 complete, valid ones, where its predecessor managed 30 on the same check. A short extra round of training on checked Forge code brought it to 27.3 before release. So even a model that's careful about people can still be shaky at building, and you only find out by checking what it builds.
How are you testing AI-built apps or AI agents before they reach your site? I'd like to hear what you check that isn't on my list.
