How to pass a json object as string to a variable

[Ramesh Ramalingam]

This question is in reference to Atlassian Documentation: Variables for deployment environments

Ask your question here...

[Steffen Opel _Utoolity_ · Accepted solution]

Update

Bamboo does not parse variable content in any way, i.e. the content is always just a string. The problem you are encountering is passing a JSON string as a parameter to AWS CloudFormation via the AWS CloudFormation Stack task, which expects those parameters to be passed into the Parameters field as a JSON string in turn, e.g.:

[
  {
    "ParameterKey": "LambdaPolicy",
    "ParameterValue": "${bamboo.lambda.LambdaPolicy}"
  }
]

When ${bamboo.lambda.LambdaPolicy} contains just the original JSON, the resulting nested JSON is invalid and triggers the encountered MalformedJsonException.

In order to pass a JSON string inside another JSON string in such a scenario, you need to escape it (e.g. manually via JSON Escape/Unescape).

Example

Given your example JSON input:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "apigateway:*"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole",
        "iam:GetServerCertificate"
      ],
      "Resource": "*"
    }
  ]
}

You would need to pass the following escaped JSON string instead (i.e. as the value of ${bamboo.lambda.LambdaPolicy} in your scenario):

{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"logs:CreateLogGroup\",\n        \"logs:CreateLogStream\",\n        \"logs:PutLogEvents\"\n      ],\n      \"Resource\": \"arn:aws:logs:*:*:*\"\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"apigateway:*\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"iam:PassRole\",\n        \"iam:GetServerCertificate\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}

This will yield the same results as passing the unescaped JSON via the AWS Management Console, where the problem does not exist due to its more advanced UI, which accepts each parameter via an individual text input field.

Here's the corresponding Bamboo log output for a variable:

Substituting variable: ${bamboo.lambda.LambdaPolicy} with {
  \"Version\": \"2012-10-17\",
  \"Statement\": [
    {
      \"Effect\": \"Allow\",
      \"Action\": [
        \"logs:CreateLogGroup\",
        \"logs:CreateLogStream\",
        \"logs:PutLogEvents\"
      ],
      \"Resource\": \"arn:aws:logs:*:*:*\"
    },
    {
      \"Effect\": \"Allow\",
      \"Action\": [
        \"apigateway:*\"
      ],
      \"Resource\": \"*\"
    },
    {
      \"Effect\": \"Allow\",
      \"Action\": [
        \"iam:PassRole\",
        \"iam:GetServerCertificate\"
      ],
      \"Resource\": \"*\"
    }
  ]
}

And here's the resulting Bamboo log output after the variable has been used inside the Parameters field:

... bamboo.custom.aws.cfn.stack.resources.tst-1-TST-AWSB-CFNPJ-21-2.parameters.LambdaPolicy: {  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "logs:CreateLogGroup",        "logs:CreateLogStream",        "logs:PutLogEvents"      ],      "Resource": "arn:aws:logs:*:*:*"    },    {      "Effect": "Allow",      "Action": [        "apigateway:*"      ],      "Resource": "*"    },    {      "Effect": "Allow",      "Action": [        "iam:PassRole",        "iam:GetServerCertificate"      ],      "Resource": "*"    }  ]}

In other words, the escaped JSON input properly ends up in CloudFormation as an unescaped JSON parameter.

Usability Improvements

I realize that this escaping requirement poses a notable usability issue (the Parameters field inline dialog examples have a a small hint that "Reserved JSON characters in Bamboo variable content need to be escaped at runtime", however, it had been missing in the task documentation, and either way this is not easily discovered). I've filed UAA-203 to contemplate how to improve the usability in this regard (suggestions welcome).

---

Initial Answer

We have outlined a more generic solution for Utoolity's Bamboo add-ons in our KB article How to supply task input from an external source like a file - here's a slightly adjusted version for your JSON use case:

Step-by-step guide

You want to inject a JSON object into a Bamboo variable:

1. Add the steps/tasks to provide the JSON object as a file, e.g. by downloading a shared artifact, checking it out from a source code repository or fetching the content from an URL with a script (the example uses the cloudformation.parameters.json Bitbucket snippet).
   
   
2. Install the Variable tasks for Bamboo add-on (still works in Bamboo 5.12, just go to 'Manage add-ons' and upload it manually).
3. Add a Variable File reader task to Get the full content of a file and inject it in a variable.
   
   
   
   1. Specify a variable name, e.g. cloudformation.parameters.
      1.  Bamboo adds the prefix bamboo, so this variable needs to be referenced as ${bamboo.cloudformation.parameters}.
   2. Specify the file provided in step 1. to read the content from.
      1. Local scope is sufficient for reusing the variable inside the same job. Read the inline help for details on Result or Plan/Deployment scoped variables.
4. Reference the variable from a subsequent task to access the contained JSON, e.g. place ${bamboo.cloudformation.parameters} into a task's input field.
   
   

Limitation

This workaround has the limitation that the included content cannot use Bamboo variables in turn, because Bamboo doesn't yet support nested/recursive expansion of variables - see https://jira.atlassian.com/browse/BAM-8266.

[Ramesh Ramalingam]

Hi Stephen,

Thanks for your reply.

I think regardless of where the inputs are coming from (file, plan variables, etc.), bamboo is still parsing the input before sending it to AWS or any other tool. So the parsing is happening after the value is supplied to the parameters. So I am not sure whether this will still work. I could be wrong but I will still give it a go. Thanks.

[Ramesh Ramalingam]

image2016-8-10 11:24:18.png

[Steffen Opel _Utoolity_]

@Ramesh Ramalingam - thank you for using our add-ons.

I've just confirmed that our resp. integration test for the referenced How to still work as documented. Accordingly, the MalformedJsonException you are encountering is hinting on this being caused by an actual JSON syntax error, specifically a missing brace ('Unterminated object') - I can think of the following possible causes:

1. the supplied JSON is indeed invalid - that's unlikely of course, but you might want to quickly verify this via e.g. JSON Editor Online
2. the supplied JSON contains another nested Bamboo variable - unfortunately this is not yet supported, see https://jira.atlassian.com/browse/BAM-8266

Hope this helps - if not, could you please make the offending JSON available so that we can try to reproduce this (and of course, feel free to create a support request, if you'd prefer to take this offline)?

[Ramesh Ramalingam]

Hi Steffen, thanks for your reply.

I was using the quotes (trail and error) to check whether the bamboo will accept the json as string or not. Obviously the json is correct as I have tested directly on AWS cloudformation and validated already even before writing the cloudformation. I have copied the JSON below.

Sorry I am not sure if I have explained you the problem clearly.

There is a parameter in AWS cloudformation which accepts string value. I am passing the json as string value for that parameter and AWS cloudformation accepts as string when I create the stack directly from AWS. But when I do that on bamboo, it determines the string as JSON while parsing and send it across as JSON to AWS instead of string. So I need to make sure I pass the below JSON as string to AWS.

{             "Version": "2012-10-17",             "Statement": [                 {                     "Effect": "Allow",                     "Action": [                         "logs:CreateLogGroup",                         "logs:CreateLogStream",                         "logs:PutLogEvents"                     ],                     "Resource": "arn:aws:logs:*:*:*"                 },                 {                     "Effect": "Allow",                     "Action": [                         "apigateway:*"                     ],                     "Resource": "*"                 },                 {                     "Effect": "Allow",                     "Action": [                         "iam:PassRole",                         "iam:GetServerCertificate"                     ],                     "Resource": "*"                 }             ]         }

[Steffen Opel _Utoolity_]

@Ramesh Ramalingam - thanks for clarifying, I think I'm on track now.

You have encountered an easy to trip over usability flaw with passing nested JSON objects via task fields that expect any input formatted as JSON in turn. The TL;DR is that you need to escape the nested JSON object before passing it inside the JSON task field - I've updated my answer with a more detailed description of the problem and solution.

As mentioned, I realize that this comprises a notable usability issue, sorry you had to discover it the hard way. Unfortunately I'm not quite sure how to address this with a reasonable amount of effort, insofar the Bamboo task UI doesn't offer any UI widgets out of the box that would allow to mirror the more advanced AWS Management Console UI. Regardless, I've filed UAA-203 to contemplate whether there might be any other options to improve on the current approach (suggestions welcome).

[Ramesh Ramalingam]

Hi Stephen, Thanks for your time again.

I did try with unescaped JSON couple of days ago and didn't work for me.

Let me give it a go again because I was adding \ next to the quotes manually. Thanks

[Ramesh Ramalingam]

Hi Stephen, It's working after I converted it through JSON Editor Online.

Thanks for your help. Appreciate your time and effort.

[Yogesh Mude[Jira]]

Hi @Ramesh Ramalingam

you can find related here JsonObject article please go through this.

might be it helps you. 

[Ramesh Ramalingam]

Thanks Yogesh. I will read through and see if I can get anything out of it. Thanks.