Recording Uploaded! How We Build AI Quality Evaluation & Improvement Framework

[#1]

Join us for an insightful virtual ACE event where we'll explore how Atlassian ensures the quality of its AI features. Our focus will be on the AI feature quality framework for Editor AI, highlighting our commitment to delivering reliable and effective AI capabilities.
Allen, one of our dedicated Machine Learning Engineers, will guide us through the intricacies of maintaining high-quality AI responses. He will share the challenges we've faced and the strategies we've developed to build and enhance our quality framework. You'll learn how we utilize data, tools, and metrics to consistently and efficiently improve the quality of our AI features.

Event Details:

• Topic: Diving Deep: AI Feature Quality Evaluation and Improvement
• Featuring: Allen, Atlassian Machine Learning Engineer
• Time: 20th Nov, 1 PM Sydney Time 

We invite you to RSVP and join us for this engaging session (link here). Bring your questions, as we look forward to an interactive discussion. If you're unable to attend, don't worry—we'll share the video on our community page for later viewing.

To stay updated on future events, consider joining the ACE Community and subscribing to the AMA chapter. We hope to see you there!

RECORDING UPLOAD:

[Byron Buckley]

Hi Allen,
Informative session.

I've noticed Editor AI is quite fast, will Atlassian allow Marketplace developers to use Atlassian models within Forge Confluence/JIRA apps?

[Rob Mkrtchian _CAIAT_US_]

Hi Allen, thanks for sharing insights into Atlassian Intelligence.

I have have question about the screen you shared during the event.

Could you please explain how AIX evaluation and prompt registry communicate with each other? And ideally share briefly what data flows into each of them...

Additionally, whether AIX evaluation is automated, manual or both?

Thanks a lot!

[Allen]

Hi ! @Rob Mkrtchian _CAIAT_US_  

prompt registry - this is a working in progress project with AI platform team, we plan to be able to:

• register - prompts, prompt meta data (parameters, model version..etc) directly from AIX eval, or many other places, and 
• load - different versions of prompts (and meta data if needed), this can be local, offline or production.

This is mimicing the idea of a "model registry" in the conventional ML world.  

Evaluation - a bit of both. Currently AIX eval is manually triggered, but we also trigger scheduled runs. 

[Allen]

Hi! @Byron Buckley  currently the Atlassian intelligence are only available for premium and enterprise customers. I don't know what's the plan on marketplace apps, but that's an interesting idea though. I can reach out and get back to you on this.