How can i parse a multi-line file of strings into a single JSON string that can be accepted by an API?

Question

I am trying to figure out how to parse a Terraform plan file in such a way that I can publish it as a comment against a commit or pull request in Bitbucket Cloud. The api endpoint I am using is here: https://developer.atlassian.com/cloud/bitbucket/rest/api-group-commits/#api-repositories-workspace-repo-slug-commit-commit-comments-post

I run terraform plan and tee the output to a file called tfplan.txt which looks something like this:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_iam_policy_document.combined_bucket_policy will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "combined_bucket_policy"  {
      + id                      = (known after apply)
      + json                    = (known after apply)
      + source_json             = (known after apply)
      + source_policy_documents = []
    }

I need to transform this output into something the api endpoint can accept. When I do this via the Bitbucket Cloud web interface, and capture the output with Inspect, its formatted something like this: (truncated for brevity)

"\n\n```\nTerraform used the selected providers to generate the following execution\r\nplan. Resource actions are indicated with the following symbols:\r\n + create\r\n <= read (data resources)\r\n\r\nTerraform will perform the following actions:\r\n\r\n # data.aws_iam_policy_document.combined_bucket_policy will be read during apply\r\n # (config refers to values not yet known)\r\n <= data "aws_iam_policy_document" "combined_bucket_policy" {\r\n + id = (known after apply)\r\n + json = (known after apply)\r\n + source_json = (known after apply)\r\n + source_policy_documents = []\r\n }

What would be an effective way, preferably in bash that i can transform the plan output into an acceptable structure to send to the api?

Thanks

Answer 1

There's a couple things you can do to simplify things here for parsing.

There's the TF_LOG_PATH variable so that you can have terraform automatically output to a file.

There's the TF_LOG variable which allows you to set the content type to JSON. This is effectively a trace but regardless, the log now becomes more easier to parse.

Once you have a parsable log file you can use the JQ utility to easily filter the things you need and build them into a new JSON string.

Once you have your new JSON string containing your message you can build the string you need to pass to curl to call the endpoint. In my experience PR messages are much simpler than commit message which would have code references.