Introducing Atlassian Data Lake for Jira Software - Early Access [NO LONGER AVAILABLE]

[#1]

Introducing Data Lake for Jira Software

We’re excited to introduce our data lake, now available in early access. It is available to preview for Jira Software Cloud Premium and Enterprise customers.

Many of our customers regularly turn to Jira reports and dashboards to learn from and improve how they work. However, some customers also require more advanced reporting capabilities that are best supported by their own BI tools. For example, some customers may need the ability to more deeply customize what they report on or they may need to combine data sets across multiple Jira sites, or with data from 3rd party tools. 

Our data lake makes it much easier to connect data from Jira Software with your BI tools so you can do just that. With the data lake, you get access to a structured data set, which is purpose built to be simple to query and simple to plug into your own BI tools. We believe this new feature will make it more accessible than ever to unlock additional value from your Jira data in your own BI tools with reports and insights tailored to your needs.

Our data lake supports popular BI tools such as Tableau, PowerBI, Looker, Qlik, Databricks, Mulesoft and SQL work bench (if the BI tool is not on this list, please note that in our sign-up form). While today our data lake is only available with Jira Software, in the future we also expect to support additional Atlassian products.

UPDATE March 2022
Thank you all for your interest in this feature. We have decided not to productionise this feature as originally designed and will be taking the architecture in a new direction to support this. It will be rolled into a more comprehensive Analytics offering included in Enterprise Edition.

More details will be provided at the Atlassian Team 22 event.

[Neeraj]

Hi Ben - Your Google form needs to have it's permissions adjusted. Getting "This form can only be viewed by users in the owner's organization.".

Looking forward to playing with this.

[Ben Jackson]

Ooops done thanks @Neeraj 

[Boris Berenberg - Atlas Authority]

Can you comment at all on how permissions will be applied to the data in the data lake?

[Ben Jackson]

Boris initially with EAP permissions are only applied on the 'access' level not at the data level. In talking to customers many said they would apply permissions to the reports and dashboards created from the data and additional granularity wasn't immediately required for some use cases.

We will look to providing more permission options over the lake data in the future but we have some internal engineering dependencies we need to wait for.

[Boris Berenberg - Atlas Authority]

I look forward to playing with this. Do you have any scoping on what data goes into the data lake in the first place? For example "exclude projects in the sensitive category".

[Darryl Lee]

One type of request that occasionally comes up is for manager/executives to see issue/ticket counts, without wanting (or needing) to dive into individual tickets. Or with siloed teams, there's been requests to actively prevent visibility into details while still offering a real-time view of high-level numbers.

Has there ever been any thoughts about how to do deal with a request like that? Would the data lake offer any opportunities to have more finely grained permissions?

[Jaswanth P]

Is it on the Atlassian roadmap to make it available on Jira data center?

[Gorka Puente _Appfire_]

Hi @Ben Jackson

We are marketplace partners but we don't have Jira Enterprise or premium, just the standard Atlassian provides to marketplace partners without badge. We are really interested in joining the program, can we?

Regards,

[Dave Liao]

@Ben Jackson - will Community Leaders be able to experiment with Data Lake, so we can help others on Atlassian Community who have questions about it? Hopefully someone's already requested the EAP for our sandbox. 🙏

[Josh Frank]

@Jaswanth P A similar feature will be available for Data Center called "data pipeline". We've included it on our public roadmap.

[Ben Jackson]

@Boris Berenberg - Atlas Authority for EAP all data from the instance will flow into the lake including any sensitive projects. Customers would then need to gate access to the BI tool using that token.

We know there are some customers that will need more granular control on the Lake data and we are looking at a number of options for that. We hope to gather feedback during the EAP to help us determine the best path forward for future replication and access controls.

@Darryl Lee as per above, we plan to add more controls as we learn from our customers. One of the options I think might be relevant is replicating all data but redacting certain data for sensitive projects/issues so its still possible to roll up the full picture.

@Dave Liao there is a Community Leader Premium instance. We will arrange for it to be provisioned into the EAP and let you know.

Any other q's let me know.

[Ben Magro]

@Jaswanth P we do have a similar feature in our Data Center offering call Data Pipeline. It's currently live for JIRA Software in 8.14.X.

Whilst Data Pipeline is not a hosted solution, it will export the relevant data to a CSV file for importation into a best of breed data hosting solution.

We are also working on bringing this to other Data Center products in the portfolio.

@Josh Frank Thanks for jumping on this one.

Ben Magro
Data Center - Product Manager

[Ben Jackson]

@Gorka Puente _Appfire_ I'll email you to discuss.

[Renee Zhang]

@Dave Liao Absolutely, Community Leaders will be able to experiment with our data lake. We ask that you sign-up here with your premium sandbox instance and we will let you know once you're in. 

[Carla Pasquali]

Hello, any plans to add this for Jira Service Management Cloud?

[Gerard Brown]

@Ben Jackson @Ben Magro @Josh Frank et. al.

I don't know if this is the direction we are moving yet.  Is there a webinar or any other introduction medium that my team can experience?

[Josh Frank]

@Carla Pasquali Yes, there are plans for Jira Service Management to follow shortly after Jira Software although we can't provide an exact timeline. The data lake is part of the Atlassian platform so will work across multiple Atlassian products. Jira Software is just the first product to be supported.

[Josh Frank]

@Gerard BrownThere's no webinar available to reference just yet, although if you're interested in participating in the early access program, we'd be happy to schedule a call with you to better understand your needs. Email us at datalakeeap@atlassian.com to connect. 

[Dorothy Krajewski]

Can you share what the details of the data in the data lake?  Do you have a data dictionary?

What I am looking for is whether the data lake will have calculated data that is presented in jira reports but not available using JQL.

Examples:

Sprint velocity: sum of issues, sum of points committed and completed in sprint

Control chart: Time in status

[Ankur Bhargava Takeda]

@Ben Jackson, we are eagerly waiting for being a part of this community, we feel we have a very strong case of using this tool in near future. Would like to know if there is any way to expedite participation process. 

Our team would be happy to get on a call with you and take some advise on how we can proceed on this.

cc @Jake Meisenbach 

[Ben Jackson]

@Dorothy Krajewski the initial lake schema does not have any preset calculations but that should be trivial for which ever BI tool you would like to connect to perform such a roll up.

@Ankur Bhargava Takeda we would be more than happy to get on a call and discuss this with you. If you'd like to send us a message via datalakeEAP@atlassian.com we can organise it.

For everyone who has registered thank you all for your interest. We have had such a big response. Provisioning is rolling out in stages so if you don't hear from our team don't be concerned we will be contacting you in the coming days.

Ben

[aneu]

@Ben JacksonWe at Manhattan associates would also like an initial call to go over what the lake contains. Are their any webinars planned? 

[Ben Jackson]

Hi Aneu yes happy to set up a call check your emails there should be a link in there to be able to book a call.

[Lisa Schaffer]

Hey @Ben Jackson - I sent an email on 2 March 2021.  I have yet to get a response.  Can someone help?

Thanks

Lisa

[Renee Zhang]

@Lisa Schaffer Hi Lisa, I've just sent you a response via email - apologies for the delay!