Has anyone found an AI review step that survives a busy sprint?

[Howard Berkman]

A few weeks ago I asked this group where AI earns its place and where you don't trust it, and the answers landed remarkably consistently: fine for drafts, summaries, and JQL; not fine for governance, compliance, or anything with real consequences.

Nobody disagreed with that line. What I still can't work out is the practical half — what people actually *do* about it.

By "survives a busy sprint," I mean it still happens in the week where the release slipped, someone's out sick, and there's a demo tomorrow. Most review habits are fine when things are calm and quietly die the first time they're inconvenient. Nobody notices they stopped.

Here's where I've landed, and I'd love to be challenged on any of it:

Asking the AI to flag its own uncertainty doesn't work. "Tell me what you're not sure about" feels responsible and doesn't track where the output is actually wrong. You get a confident answer with a confident caveat.

The dangerous failure is omission, not error. Wrong sentences get caught. A sprint summary where every sentence is true and the one blocker that mattered isn't mentioned — that goes straight to stakeholders. Nobody proofreads for absence.

Structural beats diligent.  Anything relying on someone remembering to be careful decays. The two that have held for me: make it cite — every claim tagged with the ticket ID it came from, so checking is clicking three links rather than rereading everything — and make it declare absence: "list what you looked for and couldn't find."

Attach it to a ceremony that already has a deadline. The report gets checked in the five minutes before review because that slot exists anyway. A standalone review checkpoint on the calendar is gone by the third sprint.

And whoever ran the prompt is the worst person to check it. They already believe it.

What I'm genuinely unsure about is durability. Has anyone had a review habit last a full quarter? Or does everyone rebuild it every few months after something slips through?

I wrote this up properly as a free course session — no signup, no email.

Full disclosure so nobody feels ambushed: it's a sample session from a paid course I'm building on my own time. That session is free and stays free, and I'd have written it regardless — this community is where the thinking came from.

Mostly, I want to hear what's actually holding YOU up.

[kh-samyak · Accepted solution]

@Howard Berkman 

Thanks for your post!

We’ve removed the external link from your post, per our Community Rules of Engagement, external links that may be promotional aren’t allowed in forum posts.

[Howard Berkman]

Fair enough on the link — happy to share the write-up with anyone who wants it, just ask.

[Howard Berkman]

A follow-up note for you above, @Viswanathan Ramachandran and @MeghnaP_LogicLemur Labs .  I'm curious what you both think, as you've engaged before.

[MeghnaP_LogicLemur Labs]

@Howard Berkman This is where I think the missing piece is telemetry around the AI work itself.

For teams using AI agents inside Jira, you can either download and track audit log periodically and analyse the telemetry or can use the app like AgentWorkLog for Jira which can help make that review process more durable by capturing things like:

• which Jira issues the agent worked on
• how long the agent worked
• when a human had to take over
• actual human quality feedback on each AI Agent run
• where AI work repeatedly needs rework

That changes the conversation from "Did we remember to review the AI?" to "What does the evidence show about how the AI and AI Agents performed?"

I especially like your point about omission. A useful governance signal could be identifying AI work that received no human feedback, required repeated intervention, or touched an issue without producing an accepted outcome.

The interesting next step, IMO, is connecting AI activity + human review + outcome at the Jira issue level. That's where AI governance starts becoming measurable rather than relying on people being diligent every sprint.

What do you think ?

[Howard Berkman]

Hi @MeghnaP_LogicLemur LabsThis is a better answer than the one I was fishing for, and it reframes the problem usefully — telemetry is passive, so it survives a busy sprint by NOT requiring anyone to remember anything. That's exactly the property I was looking for and couldn't name.

"AI work that received no human feedback" is the sharpest signal in your list. It's a proxy for unreviewed work that doesn't depend on anyone self-reporting whether they reviewed it, which is where every honour-system process I've tried has quietly failed.

One thing I see often, and I'd like your view because it's the part I can't solve: telemetry sees what the agent did. Omission is what it didn't do. If a sprint summary never surfaces the blocker sitting in a comment on a subtask, there's no run to flag, no rework, no intervention — the log looks clean because, from the agent's perspective, nothing went wrong. It only becomes visible when someone who already knew about the blocker reads the output.

So it feels like measuring agent performance and catching agent blind spots might be two different problems. The first you can instrument. The second seems to need a source of truth independent of the agent — which in practice is a human who knows the domain, which is the thing we were trying not to depend on.

Unless the signal is indirect: does an agent that keeps needing intervention on a given issue type also tend to be the one missing things there? If rework frequency correlates with omission, telemetry gets you a warning light even if it can't see the specific gap.

I'm curious whether you've seen that pattern in the data, or whether omission just doesn't show up at all.