Use Jira Cloud Automation to get the users e-mails

[#1]

Nowadays personal data protection is the cornerstone of many new features, released in software products. Be it GDPR, ISO 27701 or some other standard, companies are obliged to follow the rules if they do not want to face severe consequences.

Atlassian has put a lot of effort in their Cloud products to assure they comply to data protection regulations, from the feature to anonymize users to restrictions in the API when fetching user details. As necessary and important that is, it also means that routine operations, like fetching a user's e-mail to enable integration with another product in your company through REST calls can become quite hard to achieve.

The Problem:

Lately we had to come up with a way to fetch Jira Cloud users logged time and present it in JSON format to be imported by Microsoft Dynamics 365 and used for billing. After some discussions with the integrators of the Dynamics 365 solution, we agreed that the user's e-mail will be used as identification of the person that logged the time, as all other options would not guarantee uniqueness. Fast forward couple of weeks, our Jira app developers created a prototype of a standalone Java app, which used Jira Cloud REST calls to fetch all users logged time for a specified month and prepare a JSON from it. That is when we realized, that the e-mails of the users could not be fetched through REST calls, no matter what we tried. The REST call would return something like:

{
    "self": "https://somecompany.atlassian.net/rest/api/3/user?accountId=123456:38935ea6-083c-466d-9056-a1e742423fc",
    "accountId": "123456:38935ea6-083c-466d-9056-a1e742423fc",
    "accountType": "atlassian",
    "avatarUrls": {
      "48x48": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:38935ea6-083c-466d-9056-a1e7424243fc/123b8336-6a15-420c-af71-f9b3196d39c4/48",
      "24x24": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:38935ea6-083c-466d-9056-a1e7424243fc/123b8336-6a15-420c-af71-f9b3196d39c4/24",
      "16x16": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:38935ea6-083c-466d-9056-a1e7424243fc/123b8336-6a15-420c-af71-f9b3196d39c4/16",
      "32x32": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:38935ea6-083c-466d-9056-a1e7424243fc/123b8336-6a15-420c-af71-f9b3196d39c4/32"
    },
    "displayName": "John Doe",
    "active": true,
    "timeZone": "Europe/Paris",
    "locale": "en_US"
  }

So we have an accountId, that uniquely identifies the user in the Cloud but does not help us in the integration, we have displayName, that cannot be used as it is not unique. We also have info about the accountType (can be "atlassian", "customer" or "app") and if the user is "active".

At this point our integration plan seemed a bit doomed, because not having e-mail would mean we cannot really identify the users on the Dynamics 365 side. As customers do not like to get as an answer "It is not possible", we had to come up with something...

The Workaround:

As Atlassian are famous by the fact, that they have hundreds (if not thousands) of feature requests for their products, for which they provide a workaround and never really fix, we also decided to go for a "quick and dirty" one to fetch user's data when you need something like e-mail. The idea is to create a Jira issue for each user and store there their e-mail as text, so that it can be read through the API and added to the JSON. The aim was to make these issues creation automated, so that the customer would not need to do anything in the process. Jira Automation provided all the needed tools to do that, so here how we did it. 

First we created a Jira project called "UserEmailMatch", which would store the issues (team-managed one), so that these issues did not interfere with the rest of the customer data in the instance. Then the following automation rule was created:

Simply put the rule would automatically run at the start of every month, delete all the issues in the "UserEmailMatch" project and then re-create them for all active users. The search for active users is done again by using REST call like:

https://somecompany.atlassian.net/rest/api/3/users/search?query=*&startAt=0&maxResults=500

Wait, what? Didn't REST calls fail to return e-mail data for the users??? If these questions just popped up in your mind, congratulations for paying attention so far!

Yes, the REST call will not give us the needed data yet, but we do not need the e-mail for now, we only need the users. Now it is time to create an issue for each user, but first we filter the ones that are not active and are not of the correct type, because we can have a Jira instance, that has 200 users of Jira Software, 10 Jira Service Management agents and 5000 portal customers. We only need to get the data of those 210 users that can actually log time, right? So here we go:

In the first block above, we iterate over the returned users list and assign the {{NextUser}} variable with the {{webResponse.body}}, here is how it looks:

Then we compare the accountType and active values from the response with our target users - "atlassian" accounts that are active. This will filter all account types like "customer" (e.g. portal users) or "app" (e.g. app service accounts). 

{Offtopic} If you are thinking now "Oh, but that is not completely right, you can have accountType "atlassian" users, that still have "Portal only" access and in this way you will include them as well in the created issues - you are completely right and probably you should try to get the new "Atlassian Cloud Organization Admin Certification" as you are really advanced admin. I will not go here into discussing the differences between Internal and External Customers in Jira Cloud, you can go and read the articles about that and try to figure it out yourself, if you dare ;).{/Offtopic}

Now that we filtered the users, it is time to create an issue for each of them, and this is how it looks:

What we do here is to create an issue in the new project, set the Summary to be the "accountId" of the user and set a custom field of type User (we create it first and get the customfield_10045 id to use here), using the ID of the user. In this way we have the Jira User set in the new issue, and all that is left now is to create one more automation rule, that will get us the e-mail of the user:

Here you can see how we fetch the user's e-mail from the User field in the issue, using smart values, and store it in the Description field. As a result we will have for each user an issue, that has the user accountID in the Summary and the user e-mail in the Description. Just make sure you restrict this rule to work only on the issues in the "UserEmailMatch" project!!!

And that is it. Now all that was left was to change our integration app to go through these issues and find the e-mail for each user by their accoundID and put it in the resulting JSON. Long live Jira Automation and quick and dirty workarounds!

[Bill Sheboy]

Hi @Boyan Angelov (Nemetschek Bulgaria) 

You do not post an image of your complete rule, and so I wonder: are those two branches in the same, scheduled trigger rule, one to delete the "user issues" and one to recreate them?  If so, have you monitored the rule branching limits and execution time to check for throttling relative to the service limits?

Kind regards,
Bill

[Darryl Lee]

Hi @Boyan Angelov (Nemetschek Bulgaria) - interesting that you say you could not get email from REST. I've tested this query:

https://MYSITE.atlassian.net/rest/api/3/user?accountId=123456:11ff11111-RESTOFMYACCOUNTID

And got this back:

// 20240227153024
// https://MYSITE.atlassian.net/rest/api/3/user?accountId=123456:11ff11111-RESTOFMYACCOUNTID

{
"self": "https://MYSITE.atlassian.net/rest/api/3/user?accountId=123456:11ff11111-RESTOFMYACCOUNTID",
"accountId": "123456:11ff11111-RESTOFMYACCOUNTID",
"accountType": "atlassian",
"emailAddress": "darryl.lee@MYACTUALEMAILDOMAIN",
"avatarUrls": {
"48x48": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:11ff11111-RESTOFMYACCOUNTID/76f52fda-5bf0-4d68-acb5-c032a63e755a/48",
"24x24": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:11ff11111-RESTOFMYACCOUNTID/76f52fda-5bf0-4d68-acb5-c032a63e755a/24",
"16x16": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:11ff11111-RESTOFMYACCOUNTID/76f52fda-5bf0-4d68-acb5-c032a63e755a/16",
"32x32": "https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/123456:11ff11111-RESTOFMYACCOUNTID/76f52fda-5bf0-4d68-acb5-c032a63e755a/32"
},
"displayName": "Darryl Lee",
"active": true,
"timeZone": "America/Los_Angeles",
"locale": "en_US",
"groups": {
"size": 6,
"items": [

]
},
"applicationRoles": {
"size": 3,
"items": [

]
},
"expand": "groups,applicationRoles"
}

 I wonder if it's because you're in the UK? I have seen error messages like:

"The query parameter 'username' is not supported in GDPR strict mode."

But per this article, I'm also able to lookup accountIds by email, as long as I URL Encode the address:

https://MYSITE.atlassian.net/rest/api/3/user/search?query=darryl.lee%40MYACTUALEMAILDOMAIN

Which returns the same info as above without group or role info.

Because man, that is a lot of work you have to do just to get an email address.

Also, are you aware of the Organization REST API that would let you grab all of the account data, including account ID, type, status, email, and last active?

https://developer.atlassian.com/cloud/admin/organization/rest/api-group-users/#api-v1-orgs-orgid-users-get

[Darryl Lee]

By the way, I just tested the Organization REST API just now and was able to successfully get a full listing of my org's users, including account_id, email, and last active.

[Boyan Angelov (Nemetschek Bulgaria)]

@Bill Sheboy Yes, the deleting of issues and recreating them is currently in 1 rule, which indeed can lead to throttle situation if you have thousands of users in the instance. In this case you can separate it in 2 different rules, one to delete and then one to recreate, by properly scheduling them. If you still get into throttle then you will have to get creative, like not using REST to get all users, but get only users with access to certain project, etc. Rules throttling can be real pain in the @@@, but with some creativity can be avoided.

[Boyan Angelov (Nemetschek Bulgaria)]

@Darryl Lee You are right that when you call the rest for your own user it will return the e-mail as well, but have you tried calling it for other users? In my case the rest I am calling returns a JSON result, that only contains e-mail for my user, for all the rest it would not have the e-mail. 

I have not tried the Organization REST, it is possible that it returns the e-mails for all the users in the org, but in my case here I do not need customer accounts anyway (I am assuming the Organization REST would work for portal customers mostly), so this would still not help.

[Darryl Lee]

Oh crap @Boyan Angelov (Nemetschek Bulgaria), sorry, you're totally right, when I make the same call for another user it doesn't include the email.

The Organization REST API for users actually includes users of all types, including: atlassian (Jira/Confluence users), customers, and apps.

Caveat: It only works for Managed Accounts (those that are in your claimed domain).

If you do have Managed Accounts I would encourage you to look into it the Organization REST API. When you eventually get the new user management experience you'll be able to take advantage of a new API org users search endpoint that lets you filter ONLY for users, not customers or apps as well as the status, like active, which I think is your ultimate goal?

[Boyan Angelov (Nemetschek Bulgaria)]

@Darryl Lee Thanks for the update, obviously I mismatched the Organization here, it is not the Customer Organizations, but the Cloud Org, which sounds promising. However the fact, that it only works for Managed accounts means, that it would only be useful in some of the cases (in our case it is not). 

Keeping fingers crossed to get something new and useful in the new API, some more filtering will definitely save some work on the Automation side as well. In the end, with data sensitive matters I guess there will always be some trade-offs and workarounds.