Seeking feedback on Rovo & Atlassian Intelligence usage metrics

[#1]

Hey! I'm Carlee, a designer at Atlassian. My team and I are looking to understand the usage insights that your org would find most valuable to see for Atlassian Intelligence and Rovo. Here are some design concepts for your consideration.

Could you please tell us your thoughts on these charts:

• Why and how they would be useful to you?

• What would make them more useful?

Also, let us know if you'd be willing for us to contact you about your responses.

Really looking forward to reading your feedback and appreciate your time!
Thanks :)

[Matthias_Hansen]

Happy for you to contact me. What is important is a view by department / team to help with adoption / proper use.

[Wong]

Really like chart 1 and 2, we would use that to show adoption. Would also love to see a chart that shows top 10 users to help us identify "power user" who we can use as AI evangelist.

[SEMBERGER, JASON R.]

graphs 1 and 2 are great.

Clarifying questions:

• Can you define unique interactions vs non-unique interactions in graph 1? is that in reference to someone selecting one of the options in the predefined dropdown list from AI vs freeform prompting?
• For Graph 1 and 2, would you have the ability to drill down to the team level?

Ideas for future graphs (I know you didn't ask for this :-))

• # of times (or % of time) a user accepts the results from AI in a user story description (I know this is granular)
• # of times a user interacts with AI over a period of time

[Ronny Katzenberger]

I like the first 3 charts - that will be useful. But I would like to expand more on the agent usage.
Being able to filter for a set of agents would be helpful to help with agent hygiene (remove duplicative agents or suggest consolidation). For example, you select Agent 1, Agent 3 and Agent 54 and compare usage. Instead of bar charts, area charts would be helpful over a more extended period to visualize adoption/growth over time.

I'm not seeing any error rate reporting in any of those charts. I have automation with agents failing at times, which would be helpful information for debugging.

[Nicolás Guzmán]

Hi,

I think the metrics are useful for controlling and tracking the real usage and the adoption inside the instance. However, I would suggest two more metrics:

• % per project where AI or Rovo features are used the most. This could help us identify teams that are embracing the solution and engage with them to enhance our knowledge base, ensuring continuous improvement in the results.
• Number of times users give a thumbs up or thumbs down to a Rovo answer

I'm also open to having a chat with you and your team about it.

[Carlee Potter]

@SEMBERGER, JASON R. 

The thinking behind unique interactions vs non-unique interactions is to provide insight regarding adoption, i.e. are many users trying this out, or is the highest usage (interactions) from return users, possibly just “power users”. 

Thanks for the ideas, we definitely want to hear them!

[Carlee Potter]

Thank you so much @Nicolás Guzmán and @Matthias_Hansen. We'll reach out as soon as we can regarding research interview opportunities.  

[Dave Lines]

Hi, thanks for getting started on this! I'm a tech writer and would really like to know how often (if at all) the "Ask AI" feature is being used to find information in our Confluence space. Would charts 1 and 2 help with that? I'd also really like to know what the most popular search terms are. Thanks in advance and feel free to contact me.