Seeking feedback on Rovo & Atlassian Intelligence usage metrics

[#1]

Hey! I'm Carlee, a designer at Atlassian. My team and I are looking to understand the usage insights that your org would find most valuable to see for Atlassian Intelligence and Rovo. Here are some design concepts for your consideration.

Could you please tell us your thoughts on these charts:

• Why and how they would be useful to you?

• What would make them more useful?

Also, let us know if you'd be willing for us to contact you about your responses.

Really looking forward to reading your feedback and appreciate your time!
Thanks :)

[Matthias_Hansen]

Happy for you to contact me. What is important is a view by department / team to help with adoption / proper use.

[Wong]

Really like chart 1 and 2, we would use that to show adoption. Would also love to see a chart that shows top 10 users to help us identify "power user" who we can use as AI evangelist.

[SEMBERGER_ JASON R_]

graphs 1 and 2 are great.

Clarifying questions:

• Can you define unique interactions vs non-unique interactions in graph 1? is that in reference to someone selecting one of the options in the predefined dropdown list from AI vs freeform prompting?
• For Graph 1 and 2, would you have the ability to drill down to the team level?

Ideas for future graphs (I know you didn't ask for this :-))

• # of times (or % of time) a user accepts the results from AI in a user story description (I know this is granular)
• # of times a user interacts with AI over a period of time

[Ronny Katzenberger]

I like the first 3 charts - that will be useful. But I would like to expand more on the agent usage.
Being able to filter for a set of agents would be helpful to help with agent hygiene (remove duplicative agents or suggest consolidation). For example, you select Agent 1, Agent 3 and Agent 54 and compare usage. Instead of bar charts, area charts would be helpful over a more extended period to visualize adoption/growth over time.

I'm not seeing any error rate reporting in any of those charts. I have automation with agents failing at times, which would be helpful information for debugging.

[Nicolás Guzmán]

Hi,

I think the metrics are useful for controlling and tracking the real usage and the adoption inside the instance. However, I would suggest two more metrics:

• % per project where AI or Rovo features are used the most. This could help us identify teams that are embracing the solution and engage with them to enhance our knowledge base, ensuring continuous improvement in the results.
• Number of times users give a thumbs up or thumbs down to a Rovo answer

I'm also open to having a chat with you and your team about it.

[Carlee Potter]

@SEMBERGER_ JASON R_ 

The thinking behind unique interactions vs non-unique interactions is to provide insight regarding adoption, i.e. are many users trying this out, or is the highest usage (interactions) from return users, possibly just “power users”. 

Thanks for the ideas, we definitely want to hear them!

[Carlee Potter]

Thank you so much @Nicolás Guzmán and @Matthias_Hansen. We'll reach out as soon as we can regarding research interview opportunities.  

[Dave Lines]

Hi, thanks for getting started on this! I'm a tech writer and would really like to know how often (if at all) the "Ask AI" feature is being used to find information in our Confluence space. Would charts 1 and 2 help with that? I'd also really like to know what the most popular search terms are. Thanks in advance and feel free to contact me.

[Kevin Pichardo-Cedillo]

Hi @Matthias_Hansen , @Nicolás Guzmán , and @Dave Lines!

Thank you for your interest in this post. Our team is starting to explore this space and would love to chat with you. Feel free to use this link to sign up. 

Looking forward to it!

Kevin

[Dave Lines]

Hi @Kevin Pichardo-Cedillo - I signed up for an interview and I'm happy to chat, but it says "Closed Beta Feedback" and we have not done a closed beta. We are using Atlassian Intelligence but not Rovo.

[SEMBERGER_ JASON R_]

@Kevin Pichardo-Cedillo I assume open betas are only available to customer who are currently licensed for Rovo? If not, I know my organization would be willing to participate in future betas. 

[Dave Lines]

Hi @Kevin Pichardo-Cedillo - I joined the zoom call and no one showed up. ???

[Kevin Pichardo-Cedillo]

Hi @Dave Lines, my sincerest apologies. The team encountered some difficulties with Zoom that have now been resolved. I've messaged you on User Interviews. 

[Kevin Pichardo-Cedillo]

@SEMBERGER_ JASON R_ Hey! Just saw your question regarding beta programs. 

Recently, our team launched a closed beta experience for organizations with Rovo. However, in the future, we may consider launching beta programs for organizations with only Atlassian Intelligence. For those, I'm more than happy to include you!

In the mean time, if you'd like to chat about usage insights and AI adoption in your org, please feel free to book a time with the team here. 

Cheers

- Kevin