How can I integrate JIRA into Snowflake?

[Gabe Kressel]

So I basically want to create an automation where newly created JIRA issues are listed in Snowflake, in some format. I've read something about using AWS lambda and AWS S3 bucket + kinesis and firehose, but is there any other way? An easier way? I would also like to know how to exactly do this with the AWS tools.

PS. I'm quite new at this stuff 😃

[MobilityStream]

HI @Gabe Kressel ,

If you are looking for an easy way to connect Jira with Snowflake, I would also recommend to look at our Atlassian Marketplace app: https://marketplace.atlassian.com/apps/1227353/snowflake-for-jira

You will be up and running in minutes and can pick which data to load such as issues, versions, projects, users, components, etc. Unlike other solutions it won't just query each API endpoint and create a messy set of tables that require modification but produces a relatively clean table structure. Also, it is optimized for Snowflake to reduces credit usage.

Kind regards,

The Mobility Team

[Meena Bhujang]

@MobilityStream - is there a lag in data pull from jira to snowflake? How scalable is the data mapping between jira and snowflake? Is there a threshold on how  much data can be pulled before performance issues start showing?

[MobilityStream]

Hi @Meena Bhujang ,

We have several ways in which data can be exported:

• Real-time: when an issue is updated in Jira it is almost instantly sent to Snowflake. This is great when you need up-to-date data. We have instances with thousands of users, constantly making updates to issues, that run in this mode. The lag is minimal. You can also configure what event type (create, update, comment, etc) triggers an update.
• Scheduled: at a specific time a full update or a subset of an update is run. We often see clients grouping them depending on their requirements (e.g, issues hourly, worklogs daily, change history weekly, assets bi-weekly, etc). There is a lot of flexibility how this can be organized. Obviously smaller updates complete faster. When an update gets kicked off it starts exporting right away in the background.
• One-time: a one-time export doing a one-time load

All the initial exports will take some time to complete. We provide an indication on percentage complete on the App Page and also provide a table within Snowflake with some telemetry.

The app can handle any amount of data. it will just take more time to complete if you have millions of issues vs just a few thousands.

You can add any field in Jira including all custom fields. In addition you can add additional dimension and fact tables. The schema that is created is described here and should be comprehensive and easy to query: https://docs.mobilitystream.com/si/export-database-schema

If you have more detailed questions, please reach out to support@mobilitystream.com. We are happy to provide more answers.

Kind regards,

The Mobility Team

[SilkeS]

This topic has been locked because the thread has become a bit dated. If you’d like to keep the conversation going or have additional questions, we encourage you to start a new topic. You can read more about necro posting (“raising threads from the dead”) in our  Community Guidelines .

Thanks!

[Jeffrey Wolfe]

Hi,

The simplest solution for you will be to use third-party tools, like Skyvia (https://skyvia.com/data-integration/replicate-jira-to-snowflake) or similar ETL tools. They allow you to automate data loading between Jira and Snowflake easily without a need to manage complex AWS components or write custom scripts.

Hope it helps

[Gonchik Tsymzhitov]

Hi @Gabe Kressel , 

so I see a few ways:

1. Own Script to fetching and loading (ETL)

2. middleware https://jira.tosnowflake.com/

Where you can find a good solution for the fetching via exist solution on marketplace to make fetching work