what is the expected behavior after an Automation for Jira outage

Bill Sheboy

Greetings community!

My hope is that someone from Atlassian who supports or develops the automation engine for Jira Cloud can respond to this question...

What is the expected behavior for rule triggering after an Automation for Jira outage?  Specifically, will the events (scheduled or immediate) that happened during the outage window:

• eventually fire the rules,
• be lost to the "bit bucket" and so not fire,
• or depends case by case on the nature of the incident leading to the outage?

I am trying to plan for customer recovery practices after an outage that impacts automation rules.

Thanks, and best regards,

Bill

wwalser

Hi Bill,

Automation engineer here. Great question. The answer, and this is probably obvious, is your last bullet point. "depends case by case on the nature of the incident leading to the outage".

That said, we can be a bit more informative than that by explaining a bit of the architecture.

I hasten to point out that none of what follows is a promise. Our architecture will change in ways that I can't foresee. We expend great effort to ensure redundancy and good alerting, so any incident which occurs is, by the nature of it being an "incident", a surprise to us.

As events occur in Jira, Automation consumes the resulting webhooks. Mostly. We also consume events from an internal piece of infrastructure which gives access to Atlassian only events. This is how Automation knows how to trigger an Automation. When these events occur, we toss them onto a queue. A worker comes along and consumes the enqueue event. This worker looks up rules which may need to be executed based on the content of the event. If rules are found, these are placed on yet another queue which has yet more working which come along and do the actual work of the automation. The most common "work" for an automation is to make a series of API calls back to jira to perform some data mutation on one or more issues.

This is, of course, a vague sketch of the architecture but as you may be able to see from that description, failure at different points result in very different outcomes:

• If a fault occurs in Jira which causes webhooks to not be sent, Automation will not have the opportunity to execute rules.
• If a fault occurs in workers or in Jira's ability to respond to our API calls, this usually means that rules become enqueued for longer but will eventually execute. Once the fault is resolved the enqueued items can be churned through rather quickly. This ability to recover is seen by the team as a positive, we have strong alerting around queue size and mean rule end-to-end time.
• Because workers are stateful only for their lifetimes and are isolated from one another, faults on individual tenants do not propagate to other tenants via automation. I can't speak for Jira's other systems as I do not have deep knowledge of them.
• Incidents in various pieces of AWS infra affect us in different ways depending on the infra. 

I hope this helps to answer your question. Our goal is to create a system which is immune to any single point of failure, which fails gracefully when it fails (doesn't propagate failures across tenants), which scales as both Atlassian grows but also as individual tenants increase their usage of automation, and which has strong alerting so that we know about incidents before customers.

Best,
Wes