Clone linked issues from JQL Branch Rule and keep them in order?

[Darryl Lee]

I swear @Bill Sheboy has answered similar questions but I couldn't figure out if any of them fit my use case.

In a nutshell, I have a bunch of tasks linked to a "parent" Epic. (They aren't actual child tasks because they're under a different Epic. It's complicated and I'll explain it in a separate article if I get this working.)

So I'm using a Branch rule / related work items with the following JQL query:

issue in linkedIssues("{{certParentKey}}") order by issuekey

So yeah, the query returns ISSUETMP-6808 , ISSUETMP-6809 , ISSUETMP-6810 , ISSUETMP-6811, ISSUETMP-7298 , ISSUETMP-7302, in that order.

The thing is, the person who requested this automation would like those tasks to be kept in the same order.

And well, since we've been waiting 5 years for actual resolution of AUTO-32, the ticket formerly known as Allow switch between parallel and sequential execution of Automation Rules ... I can't figure out how to do this.

To paraphrase Princess Leia... "Help me @Bill Sheboy you're my only hope"

% telnet telehack
.starwars

Alternately: https://www.asciimation.co.nz/

[Bill Sheboy · Accepted solution]

Hi @Darryl Lee 

Until a sequential branching feature is added to rules, there are a few workarounds, each with their own cost / benefits / risks.  Some of these will not work with linked work items, but I will list all the ones I can think of regardless.

Some shared risks for the workarounds are:

• although there is an upper limit on links to a work item (2000, I think), a solution to handle 0-to-many for cloning may lead to rules exceeding several service limits
• there are no built-in Jira constraints to prevent linkage cycles; this could cause an endless loop for walk-the-chain scenarios (e.g., A->B, B->C, C->A...oops!)
• any rule which does several actions in a row could fail in ways difficult to clean-up
• rule outages may obscure what did / did not happen, making "catching up" challenging
• maintaining complicated rules is challenging and brittle for teams

#1) One-by-one creates

• With a limited and known number of work items, add them one by one with actions
• When a rule uses a single-item branch, that is run inline and can work around parallel processing (e.g., JQL branch on key = ABC-123)
• Please test thoroughly!  I recall the behavior of inline execution of single-item branches changed early after Code Barrel was acquired by Atlassian.  As this behavior is undocumented, there is no guarantee it will not change in the future.

#2) Chained rules (FYI -- I would not recommend trying this one.)

• assuming fewer than 10 linked work items...(and a lot of hope)
• create a rule which identifies and clones the first linked work item, adding a "magic" value to it (e.g., custom field at the same time as the create)
• a second rule, triggered on work item create, and with "Allow Rule Trigger" enabled, detects the magic value, identifies the next work item to clone**, and does so
• that ** part above is left as an exercise for the reader, and likely involves dynamic JQL and list searching shenanigans techniques 

3) Multiple rules with recursion

• First rule gathers the work items to process, and then calls another using Send Web Request, passing the data needed in the request
• Second rule uses the Incoming Webhook Trigger, processes one item, removes it from the list, and when more remain, calls itself with Send Web Request
• RISKS: there is no documentation as to whether or not this approach is checked with the looping service limit although that could be validated with a couple of test rules to check if a header (or whatever) is added behind the scenes with Send Web Request to detect the request source...leading to the limit check failure
• The general approach is outlined in this Atlassian team member article 

#4) Build your own external app

• create a service in a language which supports sequential loop processing
• call that with Send Web Request

#5) Build your own Forge action???

• I have not tried this, but perhaps a Forge app could be created to build a new rule action to sequentially add work items
• I suspect a marketplace vendor may create / sell such a thing, although I have not searched for one

#6) Investigate marketplace apps / addons

• I recall there are several cloning-related apps, and perhaps one fits your scenario with the linked work items

Good luck, Darryl, and let us know what you try.  Thanks, sir!

Kind regards,
Bill

[Darryl Lee]

Thanks for responding to the Bat-Signal, @Bill Sheboy (truly mixing the movie/tv references), I knew you'd come through.

#1) Alas, no, the number of linkedIssues is variable, depending on the type of certification

#2) My reaction while reading this was that I said "Oh god" then made this face:

(I was gonna use clip art, but eh, cool hat.)

#3) Hmmmmmaybe? I mean should the fact that an Atlassian team member wrote the article mean it's kinda sorta supported? Hahaha. Ok, yeah, I guess it's like an internal function instead of doing #s 4 or 5...

#4) Ah right, like a AWS Lambda job that accepts list of issues, and new parent, and then serially creates them? That doesn't sound awful, except the maintenance part.

#5) Oh god. Yeah, this is what @Caterina Curti and @Bryan Guffey would surely suggest. Also maintenance, but I guess at least I wouldn't have to worry about an AWS bill on top of it. (As miniscule as that would be.)

#6) We're too cheap for an app, and it also feels like overkill, but I'll take a look. Geez, there are so many. Maybe we really do need Personalized app recommendations in search results! (Kidding. I hate that idea.)

Ok, lots to chew on. Thanks!!

[Bill Sheboy]

Hi @Darryl Lee -- how about a maybe #7 approach: 

Using the Bulk Create Issue endpoint, use list iteration over a lookup result for the linked work items to build the JSON message, and try adding them with one Send Web Request.

I could not find information if the items are iterated to add sequentially or in parallel, so testing would be required.  And, this puts the entire burden on the rule-writer to manage all the cloned fields in the message.

[Darryl Lee]

Woof. I think there's a couple of issues with that. It's Bulk Create, not Bulk Clone. Reading the (admittedly sparse) docs, I don't see a way to reference a "source" issue to copy.

So I'd have to get pass it the full content of each child task I'm cloning? (I mean it's probably just Summary and Description, but what if I miss a field?)

But I'm now leaning towards #3 because despite the scariness of recursion, I do like the idea that everything stays within Automation. No external calls, no API calls. It's all blocks, so if somebody does have to figure out what the hell I did, they won't find some black box somewhere outside of Jira.

Nope, it'll be a black box in another rule. :-}

[Darryl Lee]

Heya @Bill Sheboy I was trying to figure out how to do #3 without Automation having a shift() function.

From the "Detect new Epic Created" rule, I'm sending this to the webhook:

{
  "parentKey": "{{newIssueKey}}",
  "childArray": {{lookupIssues.key.asJsonStringArray}}
}

And in the "Create Child Function" rule I'm doing:

parentKey = {{webhookData.parentKey}}
childArray = {{webhookData.childArray}}

So I end up with

parentKey: ITTEST-182

And

childArray: ISSUETMP-6808, ISSUETMP-6809, ISSUETMP-6810, ISSUETMP-6811, ISSUETMP-7298, ISSUETMP-7302

But weirdly I had to do a split() to get the first child to not come up empty:

thisChild = {{childArray.split(",").first}}

To return:

thisChild: ISSUETMP-6808

So the last bit though is trying to remove it the first element so I can call the rule again and start the recursion train.

I've tried to set remainingChildren to:

{{#childArray}}
   {{#if(not(equals(thisChild)))}}{{.}}{{/}}{{^last}},{{/}}
{{/}}

But I just get ALL of the issues, although there is a trailing comma:

remainingChildren: ISSUETMP-6808, ISSUETMP-6809, ISSUETMP-6810, ISSUETMP-6811, ISSUETMP-7298, ISSUETMP-7302,

So yeah, weird I guess I need a split again:

{{#childArray.split(",")}}
   {{#if(not(equals(thisChild)))}}{{.}}{{/}}{{^last}},{{/}}
{{/}}

So no trailing comma (so yeah, need the split), but also my not equals test seems to be failing:

remainingChildren: ISSUETMP-6808, ISSUETMP-6809, ISSUETMP-6810, ISSUETMP-6811, ISSUETMP-7298, ISSUETMP-7302

I took a wild stab with match(), but nope:

{{childArray.not.match(thisChild)}}

That returns nothing.

Any ideas?

[Darryl Lee]

Oh god. World's jankiest solution:

childComma = {{thisChild}},

And then:

remainingChildren = {{childArray.remove(childComma)}}

Success!

ISSUETMP-6809, ISSUETMP-6810, ISSUETMP-6811, ISSUETMP-7298, ISSUETMP-7302

YEESH.

I recall you writing something (and filing a bug?) about how Automation is pretty flexible/lazy about keeping lists actually as lists, hence the need to keep splitting them. I guess having it as text has its advantages. 

Now I guess I need to split it and call itself again with this payload

{
  "parentKey": "{{parentKey}}",
  "childArray": {{remainingChildren.split(",").asJsonStringArray}}
}

OH crap. Forgot that the last element will not have a comma. Dangit, runaway recursion train!

I will figure that out tomorrow, *sigh*.

[Darryl Lee]

Oh oh, I got it:

thisChildRegex = {{thisChild}},?

(Where ,* means 0 or 1 commas.)

And then:

remainingChildren = {{childArray.replaceAll(thisChildRegex,"")}}

[Bill Sheboy]

@Darryl Lee there are several ways to remove the first item from a list using smart values, including using the {{^first}} ... {{/}} handling once inside of the long-format iteration.

For example:

{{#varList.split(",").trim}}{{^first}}{{.}},{{/}}{{/}}

Notice this will add each item after the first one with a trailing comma.  That result may be stored in a variable, and then remove the last comma with the function substringBeforeLast(",").

[Darryl Lee]

Oh my gosh, I didn't even consider that {{^first}} could be used to lop off the first element, duh.

Couldn't you avoid the trailing comma with the old {{^last}},{{/}} trick? (One of the few things Atlassian actually documents?)

Let's see! Ah yeah, this is much cleaner (albeit a bit more cryptic than my original solution) (And yes, I am overconfident that a JQL Lookup -> WebRequest will not introduce extraneous whitespace, so I dropped the trim):

{{#childArray.split(",")}}{{^first}}{{.}}{{^last}},{{/}}{{/}}{{/}}

Any ideas on a more elegant way to see when I'm "done" (no more remainingChildren)? Last night I settled upon this:

{{remainingChildren.length()}} does not equal 0

Ok I finally found one of the times you very aptly explained the whole created variable/list/array situation:

Created variables are text, and when the web response is stored it loses all name attributes and typing information.  And so concatenating the values just produces a longer text string. 

(Jira Automation list doesn't work)

Ah, found the ticket I was looking for:

JIRAAUTOSERVER-1047 - "Create variable"- Create Type Variable action to preserve the source data typing 

I take your point:

Changing the behavior of Create Variable would dramatically and negatively impact a vast number of customers' existing rules.  Perhaps the suggestion could be improved: please add a Create Typed Variable action to preserve the source data typing.

But at the same time, K-IT Team makes a very good point:

The list also gets forcibly converted to a string/text - which makes using it later very cumbersome, as you will have to use .split() all the time to force the string to be treated as a list again.

And finally, that conversion happens silently and very non-transparent, it is quite hard to discover this happens. So at the very least the ‘Create variable’ action should contain a warning message that informs the user about the fact that anything will be converted to text/string type.

But yeah, I guess the documentation is pretty clear:

Note that this will always return a string. For example, if you created variable called Stakeholders with the smart value {{issue.watchers.distinct}}, your variable would return the list of watchers but the list would be rendered as a string.

(Jira automation actions - Create variable)

I guess I just have to lock this mantra into my head: "variables are strings, variables are strings" and "you always have to split variables that contains a list" Because late last night I tried to do:

childArray = {{webhookData.childArray.split(",")}}

And it returned:

childArray: [ISSUETMP-7302], [ISSUETMP-7298], [ISSUETMP-6811], [ISSUETMP-6810], [ISSUETMP-6809], [ISSUETMP-6808]

And I realize now it somehow converted a list into list of 1-element lists. Oy.

ANYWAYS, I'll post the completed rule in a bit, but thanks so much for the help!

[Darryl Lee]

Gah. It seems that I can't use a Space KEY in Clone work item.

I have passing in the {{parentKey}} and I'm extracting parentSpace thusly:

{{parentKey.split("-").first}}

And Audit Log shows: parentSpace: ITTEST

Clone work item lets me use a Smart value - cool cool cool.

But then I get this error:

Unable to create work item

                Specify a valid project ID or key (project)

Issue was created successfully, however some unexpected errors occurred -

                INVALID_COMPONENT_CONFIGURATION

AUGH. I even tried adding .trim (thanks @Bill Sheboy) and .toLowercase() but even with {{parentKey.split("-").first.trim.toLowerCase()}} -> ittest, it fails with the same error.

Since the error message mentioned project ID, I dug up and then hard-coded {{parentSpace}} to the Project ID. And... WTH, it works. Ugh, so now I need to do a freaking API call just to get the project ID? Booo.

[Bill Sheboy]

Yup: gotta use the project ID rather than the key when one wants to dynamically select the project in a rule action such as clone, create, etc.

I wish there was a clone work item endpoint to simplify such scenarios.  Better still, a clone-work-item-tree-with-children-whilst-preserving-the-order-of-children endpoint :^)

[Darryl Lee]

Sounds like this is a known issue? Was it mentioned in a previous answer? Of course Atlassian doesn't have documentation, and the error message specifically says:

                Specify a valid project ID or key (project)

Grrr...

[Bill Sheboy]

Here the link to the suggestion to support project keys and ID values, including a link back to the 2018 community post when Code Barrel's team acknowledge the limitation:

https://jira.atlassian.com/browse/AUTO-839

https://community.atlassian.com/forums/Jira-questions/Use-a-Field-Smart-Value-For-Destination-Project-when-Creating/qaq-p/822356

[Darryl Lee]

Thanks @Bill Sheboy - this actually led me to a workaround from having to make a Web Request to lookup project ID, by way of @Haddon Fisher's solution of setting the project in the "Additional Fields" JSON. As he writes:

Ticket Creation
Use "same project" value in the "Create Issue" action and then use the following JSON in the "Additional Fields" section:

{
  "fields": {
    "project": {
      "key": "{{project}}"
    }
  }
}

This worked nearly perfectly. I say nearly because it turns out that if you do specify a Project from the dropdown or {{projectId} (but not {{projectKey}}) during the Clone work item, Automation is ok with a field not being unavailable in the target project.

Unfortunately if you set the Project via "Additional Fields", and the issue you are cloning includes a "missing field", you get this error:

Field 'customfield_10291' cannot be set. It is not on the appropriate screen, or unknown. (customfield_10291)

Again, if you specify Project by a {{projectId}} Smart Value, or you are selecting it from the dropdown, no problem. Weird, and kind of a bummer. (In our use case, the field wasn't required at the end, so we just deleted the values.)

Yes, I promised a full write-up. Tomorrow, for sure. :-}

[Darryl Lee]

SIDENOTE: @Haddon Fisher I love your site name, and your mission statement:

I will be digging deeply into Jiramancy very soon!

[Haddon Fisher]

This is definitely my "garage band dad getting a shout-out from Bruce Springsteen" moment. ::tries to play it cool::

Full disclosure, I:

• Am working on my phone so I don't have access to Jira to actually test this theory...I can't remember how clone works and I am worried about how to feed it an issue in a branch or how you'd cycle through the list if you're in a branch already.
• Was up until about 1am last night confirming that Automation's `isEmpty()` function returns a `FALSE` when it's false but a ` ` when it's true, so my brain no work good.

...but @Darryl Lee could you take the results of your JQL and feed the keys into a lookup table or dynamic lookup table and then iterate the cloning process through that list? I was able to do something sort of similar here. If I get any time this weekend I will give this a whirl and see if I can figure out why it doesn't work.

[Darryl Lee]

Hey @Haddon Fisher no need for modesty. I've seen your distinctive avatar all over these forums along with your posts. And man, that Jiramancy!

We are all garage band dads, and we are all Bruce Springsteen. And Spartacus.

Yeah, in the first iteration of this rule, it was just one rule, and doing it in a JQL branch introduced the parallel problem.

I'm pretty sure that putting them into a lookup table would still mean that I'd ultimately run it through via Advanced branching through the array/list and I'd still run into the ordering problem:

Ordering of branch executions

• Branches on multiple issues (such as 4 sub-tasks) will run in parallel with no guarantee one will finish before the next one starts. Therefore, you cannot rely on changes between branches.

• Branches on multiple issues are run as a new process, with the main branch continuing execution before the sub-branch starts.

So yeah, that's why I ended up following Bill's suggestion of a recursive rule that does the equivalent of a shift() to get each issue to clone IN ORDER, and then calls itself with the shortened list each time. 

Re: FALSE vs ' '

Ooof, I don't recall what post it was but I remember the first time I saw @Bill Sheboy use {{some.function|0}} and I was like... "What is THAT", and I think it had to do with whole false/empty/null issue and man, it SUCKS working in an environment that is so weirdly and annoyingly inconsistent.

Oh here's an answer where he touches upon the inconsistency of checking field values.

[Darryl Lee]

Oh dur, I didn't look at your example. So the problem with is that is the number of child tasks is variable.

I guess the new Dynamic Lookup Tables could help me out here. Oh, I can access the index... I could do:

Table variable name = childissue

Dynamic Souce = {{lookupIssues}}

Define Entries:

key = index

value = key

So I guess I could just put as many clone issue blocks as the maximum # of children for any template and then do:

IF childissue.get("1") IS NOT EMPTY

CLONE childissue.get("1")

IF childissue.get("2") IS NOT EMPTY

CLONE childissue.get("2")

...

IF childissue.get("max#") IS NOT EMPTY

CLONE childissue.get("max#")

But ew. That's how I used to write code before I learned how loops work. :-}

On the other hand, this would be self contained all in one rule. No funky recursion, also no need for your JSON hack, because I the project for each clone would be Same as Trigger Issue.

Maybe I'll mess with it!

[Hartik · Accepted solution]

you can’t reliably keep order when cloning from a JQL branch. Automation runs branch items in parallel, so even if the JQL is sorted, the clone actions don’t respect that order.

The only practical workaround is to add an explicit ordering field first, like a number or rank copied into a custom field, then sort the cloned issues afterward using that field. If order truly matters at creation time, Automation can’t do it today and AUTO-32 is still the blocker.

[Darryl Lee]

Thanks - good point on ranking after creation, although what they really want is to see those child tasks in order while viewing the epic.

Hum, will need to look at @Bill Sheboy 's suggestions.