Automating UPM using API's - PART 2

[#1]

Continued from Part 1...

Downloading from Marketplace

Downloading from the marketplace is relatively straight forward. There are a number of API calls for each app to the Marketplace API in order to obtain all the information required. I've found using a local YAML file that contains the list of apps and the version you wish to install to be useful.

Tip: Remember to change the app ID's and version you want to be installed in the YAML file

Add to the bottom of the python file the following, which will load the YAML file, and then based on the arguments passed when you run the file, perform the activity you requested

# Load YAML file
file_name = "my-atlassian-apps.yml"
with open(file_name) as f:
    yaml_file = yaml.safe_load(f)

artifactory_url = yaml_file['atlassian']['artifactory_url']
marketplace_url = yaml_file['atlassian']['atl_marketplace_url']
marketplace_cdn = yaml_file['atlassian']['atl_marketplace_cdn']


def atlassianAPI(host, url, name=None, request=None, payload=None, params=None):
    print(f"{Style.BRIGHT}{name}") if name is not None else None
    if request is None:
        request = "get"
        headers = {
            'Content-Type': "application/json",
            'Authorization': "Basic " + b64authtoken
            }
    else:
        headers = {
            'Content-Type': "application/vnd.atl.plugins+json",
            'Authorization': "Basic "+b64authtoken
            }
    r = requests.request(request, f"{host}/rest/{url}", json=payload, headers=headers, params=params, verify=False)
    if r.status_code == 401:
        print(f"{Fore.RED}Wrong username/password")
    if r.status_code == 404:
        print(f"{Style.DIM}App not installed")
        return
    if r.status_code != 200 and r.status_code != 201 and r.status_code != 204:
        print(f"{Fore.MAGENTA}Something went wrong.)
        print(r.status_code, host, url)
        quit()
    return r.json()


def artifactoryAPI(host, url, name=None, payload=None, params=None):
    request = "get"
    r = requests.request(request, f"{host}/api/{url}", json=payload, params=params, verify=False)
    if r.status_code == 401:
        print(f"{Fore.RED}Wrong username/password")
        quit()
    if r.status_code != 200 and r.status_code != 201 and r.status_code != 204 and r.status_code != 404:
        print(f"{Fore.MAGENTA}Something went wrong.")
        print(r.status_code, host, url)
        quit()
    return r.json()


def downloadApp(app, file_path, url=None, bin_file=None):
    if url is not None:
        marketplace_version = (atlassianAPI(host=marketplace_url, url=url))
        marketplace_download = marketplace_cdn + marketplace_version['_links']['artifact']['href'].replace('rest/2/assets/', '').replace('%2F', '/')
        filename = str(marketplace_download.split('/')[6])
    if bin_file is not None:
        marketplace_download = bin_file filename = str(bin_file.split('/')[7])
    filepath = pathlib.Path(f"{file_path}/{filename}")
    print(f"{Style.BRIGHT}{appkey} - version: {appdata['version']}")
    if not filepath.is_file():
        with open(filepath, "wb") as f:
            r = requests.get(marketplace_download, verify=False, allow_redirects=True, stream=True)
            f.write(r.content)
    else:
        print(f"Is already downloaded: {Fore.GREEN}yes")
    return filename


def upmToken(host):
    request = "get"
    headers = {
        'Content-Type': "application/vnd.atl.plugins.installed+json",
        'Authorization': "Basic "+b64authtoken
        }
    r = requests.request(request, f"{host}/rest/plugins/1.0/?os_authType=basic", headers=headers, verify=False)
    return r.headers['upm-token']




for environment in yaml_file['atlassian']['environments']:
    if environment['name'] in args.environment:
        for stack in environment['stacks']:
            if stack['app_stack'] in args.stack:
                # Download all required apps for each environment
                if args.download:
                    os.makedirs(f"{args.application}/{stack['app_stack']}", exist_ok=True)
                    host = stack[args.application]['app_url']
                    for appkey, appdata in stack[args.application]['applications'].items():
                        if appkey != "jira-servicedesk":
                            downloadApp(app=appkey, file_path=f"{args.application}", bin_file=f"https://atlassian.com/software/{args.application}/downloads/binary/atlassian-{appkey}-{appdata['version']}-x64.bin")
                        else:
                            downloadApp(app=appkey, file_path=f"{args.application}", url=f"2/addons/com.atlassian.servicedesk.application/versions/name/{appdata['version']}")
                # Upload all required apps for each environment to Artifactory
                if args.upload:
                    artifactory_result = (artifactoryAPI(host=artifactory_url, url=f"storage/ATL/apps/{args.application}/{api_result}"))
                    if artifactory_result.get('errors'):
                        uploadPath = ArtifactoryPath(f"{artifactory_url}/ATL/apps/{args.application}")
                        uploadPath.deploy_file(f"{args.application}/{stack['app_stack']}/{api_result}")
                        print(f"Uploaded to Artifactory: {Fore.YELLOW}yes")
                    else:
                        print(f"Is already uploaded: {Fore.GREEN}yes")

Note: This assumes you have a repo in Artifactory called: ATL and that you don't require a username/password to upload to it...

That's it - run the following to automatically download from the marketplace and upload to Artifactory ready to install locally.

python atlassian-apps.py -e prod -s primary -a jira --download --upload

Installing to UPM from Artifactory

Installing is the fun part, but note there isn't an easy way to know once an app has been installed (or if there is, please let me know :D)... so we use a lazy sleep option.

Add this after the above code but in line with the first "IF args.download":

# Install all required apps for each environment from Artifactory
if args.install:
    host = stack[args.application]['app_url']
    for appkey, appdata in stack[args.application]['apps'].items():
        api_result = (downloadApp(app=appkey, file_path=f"{args.application}/{stack['app_stack']}", url=f"2/addons/{appkey}/versions/name/{appdata['version']}"))
        check_exists = (atlassianAPI(host=host, name=None, url=f"plugins/1.0/{appkey}-key"))
        if not check_exists or check_exists['version'] != appdata['version']:
            print(f"Version installed: {Style.DIM}{check_exists['version']}") if check_exists else None
            payload = dict(
                pluginUri = f"{artifactory_url}/ATL/apps/{args.application}/{api_result}"
                )
            api_upmToken = (upmToken(host=host))
            install_app = (installApp(url=host, payload=payload, request="post", upm_token=api_upmToken))
            print(install_app['status'])
            time.sleep(60)

And that's it - every 60 seconds it will send a new install app command to the UPM and iterate until you have no more apps to install (if the app is already installed, and the same version as the app you're trying to install, it will skip it)

python atlassian-apps.py -e prod -s primary -a jira --install

Now that you've installed your apps, let's license them!

Licensing your apps via UPM

This part is quick and painless. It will save you hours in the long run, especially if you have multiple test environments you want to swap licenses out (i.e. prod licenses with developer licenses for your non-prod stacks)

In the below example, it uses a local JSON file that contains all your licenses. The JSON file looks something like:

{
  "jira-servicedesk": "license_goes_here",
  "jira-software": "license_goes_here",
  "co.uk.jackgraves.jira-optimiser": "license_goes_here"
  etc
}

You could easily swap this out for something like AWS SecrestManager. 

Again, add this after the above code, and in line with "IF args.download" and "IF args.install":

# Retrieve licences for application and apps, and then apply them
if args.license:
    # Load JSON file
        file_name = f"{args.application}-license.json"
        with open(file_name) as f:
            j = json.load(f)
    host = stack[args.application]['app_url']
    print(f"{Style.BRIGHT}{Fore.BLUE}Checking {host}:")
    if args.application == "jira":
        for appkey, appdata in stack[args.application]['applications'].items():
            api_result = (atlassianAPI(host=host, name=appkey, url=f"plugins/applications/1.0/installed/{appkey}"))
            if api_result['license'].get('rawLicense') != j.get(f'{appkey}'):
                payload = dict(
                    licenseKey = j[f'{appkey}']
                    )
                api_result = (atlassianAPI(host=host, name="installing license", url=f"plugins/applications/1.0/installed/{appkey}/license", request="post", payload=payload))
                print(f"{Fore.GREEN}License updated!")
            elif
                api_result['license'].get('rawLicense') == j.get(f'{appkey}'):
                print(f"{Style.DIM}License already up-to-date")


    if args.application != "jira":
        print(f"{args.application} licensing is not supported at this time, however I can license your apps for you :)")
    # License apps
    for appkey, appdata in stack[args.application]['apps'].items():
        api_result = (atlassianAPI(host=host, name=appkey, url=f"plugins/1.0/{appkey}-key/license"))
        if api_result.get('rawLicense') != j.get(f'{appkey}') and j.get(f'{appkey}'):
            payload = dict(
                rawLicense = j[f'{appkey}']
                )
            api_result = (atlassianAPI(host=host, url=f"plugins/1.0/{appkey}-key/license", request="put", payload=payload))
            print(f"{Fore.GREEN}License updated!")
         elif api_result.get('rawLicense') == j.get(f'{appkey}') and j.get(f'{appkey}'):
             print(f"{Style.DIM}License already up-to-date")
         else:
             print(f"{Style.DIM}No license required")

And that's it! you're done!

Now to add the license, simply run:

python atlassian-apps.py -e prod -s primary -a jira --license

Coming soon - I'll be making these available via Bitbucket/GitHub in the coming future. 

Hope you enjoyed, let me know if you have any questions.

- JiraJared!

[JiraJared]

Part 1 can be found here: https://community.atlassian.com/t5/Jira-articles/Automating-UPM-using-API-s-PART-1/ba-p/1367368

[jesse_j_borden_ctr]

@JiraJared Did you ever get to adding these to GitHub? I think this is good, but there are some copy paste syntax issues with using this page to copy.

[JiraJared]

@jesse_j_borden_ctr  hello, Thanks for reading my post! 
I’ll share a link shortly - I didn’t get around to doing it as what a year 2020 was :/