Cannot use variable inside json --data

Question

I have a working API (linode) to update a domain. So, below is a normal API call that work just fine to update data:

  #!/bin/bash

  # hardcoded data for testing
  LINODE_API_KEY=1234
  domain_id=1931316
  domain_name="abx.com"
  domain_type="master"
  domain_email="hello@abx.com"
  domain_ttl=30

  
  # Update the domain
  curl -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${LINODE_API_KEY}" \
  -X PUT -d "{

  \"domain\": \"${domain_name}\", \"type\": \"${domain_type}\", \"soa_email\": \"${domain_email}\", \"ttl_sec\": ${domain_ttl}
  
  }" "https://api.linode.com/v4/domains/${domain_id}"

When I executed the above update API, it's working fine and I get the json response as:

{"id": 1931316, "type": "master", "domain": "abx.com", "tags": [], "group": "", "status": "active", "errors": "", "description": "", "soa_email": "hello@abx.com", "retry_sec": 0, "master_ips": [], "axfr_ips": [], "expire_sec": 0, "refresh_sec": 0, "ttl_sec": 30, "created": "2022-12-13T09:01:01", "updated": "2022-12-14T03:26:27"}

But the problem is I want to use variable inside the data. So I change the working code above into this (with extra data that I want to pass):

  #!/bin/bash
  # hardcoded data for testing
  LINODE_API_KEY=1234
  domain_id=1931316
  domain_name="abx.com"
  domain_type="master"
  domain_email="hello@abx.com"
  domain_ttl=30
  
  # This value must be provided first
  if [ -z "${domain_name}" ]; then
     echo "You must provide domain name"
     exit 1
  else
     change_domain_name="\\\"domain\\\": \\\"${domain_name}\\\""
  fi      

  if [ -n "${domain_type}" ]; then
     change_domain_type=", \\\"type\\\": \\\"${domain_type}\\\""
  else
     change_domain_type=""
  fi

  if [ -n "${soa_email}" ]; then
     change_domain_email=", \\\"soa_email\\\": \\\"${domain_email}\\\""
  else
     change_domain_email=""
  fi

  if [ -n "${domain_ttl}" ]; then
     change_domain_ttl=", \\\"ttl_sec\\\": ${domain_ttl}"
  else
     change_domain_ttl=""
  fi
  update_string="${change_domain_name}${change_domain_type}${change_domain_email}${change_domain_ttl}"

  # Update the domain
  curl -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${LINODE_API_KEY}" \
  -X PUT -d "{

  ${update_string} # THE PROBLEM IS HERE WHEN USING THIS VARIABLE
  
  }" "https://api.linode.com/v4/domains/${domain_id}"

The API will complain about Invalid JSON

{"errors": [{"reason": "Invalid JSON"}]}
  

But when I use echo ${update_string} variable, I will get the same data syntax like I use so why does it complain this is invalid?

I can even copy paste back the echo result above inside the -d data and it's working fine.

I rolled back my question to the original as to explain why I use the above method instead of creating jq --arg as suggested @Arnaud Valmary.

So for example here I only want to pass value domain_name and domain_ttl. So the value of the update string would be like this:

update_string="${change_domain_name}${change_domain_ttl}"

where the others were empty. So, I'm not sure how to achieve this using jq --arg

In brief, if domain_type="" or domain_type is empty, I don't want this variable to be in the --arg options, so user have a choice not to update this value.

Answer 1

It's better to build JSON data with a dedicated tool like jq . Bash string construction is hazardous.

#! /usr/bin/env bash

# hardcoded data for testing
declare LINODE_API_KEY=1234
declare domain_id=1931316
declare domain_name="abx.com"
declare domain_type="master"
declare domain_email="hello@abx.com"
declare domain_ttl=30

declare json_query_data='{}'
# This value must be provided first
if [ -z "${domain_name}" ]; then
    echo "You must provide domain name"
    exit 1
else
    # shellcheck disable=SC2016
    json_query_data=$(jq --arg domain "${domain_name}" '.domain |= $domain' <<<"${json_query_data}")
fi

if [ -n "${domain_type}" ]; then
    # shellcheck disable=SC2016
    json_query_data=$(jq --arg type "${domain_type}" '.type |= $type' <<<"${json_query_data}")
fi

if [ -n "${domain_email}" ]; then
    # shellcheck disable=SC2016
    json_query_data=$(jq --arg soa_email "${domain_email}" '.soa_email |= $soa_email' <<<"${json_query_data}")
fi

if [ -n "${domain_ttl}" ]; then
    # shellcheck disable=SC2016
    json_query_data=$(jq --argjson ttl_sec "${domain_ttl}" '.ttl_sec |= $ttl_sec' <<<"${json_query_data}")
fi

# Update the domain
curl -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${LINODE_API_KEY}" \
    -X PUT \
    -d "${json_query_data}" \
    --url "https://api.linode.com/v4/domains/${domain_id}"

cURL command is:

curl -H 'Content-Type: application/json' -H 'Authorization: Bearer 1234' -X PUT -d '{"domain":"abx.com","type":"master","soa_email":"hello@abx.com","ttl_sec":30}' --url 'https://api.linode.com/v4/domains/1931316'

Answer 2

This is an error-prone endeavor. You're better off using a JSON-parser like xidel to build your JSON data:

$ xidel -s \
  --variable name="$domain_name" \
  --variable type="$domain_type" \
  --variable email="$domain_email" \
  --variable ttl="$domain_ttl" \
  -e '{
    "domain":$name,
    "type"?:if ($type="") then () else $type,
    "soa_email"?:if ($email="") then () else $email,
    "ttl_sec"?:if ($ttl="") then () else int($ttl)
  }'
{
  "domain": "abx.com",
  "type": "master",
  "soa_email": "hello@abx.com",
  "ttl_sec": 30
}

If only $domain_name was declared, then in this case only {"domain": "abx.com"} would be returned.

Curl

Generate the (serialized) JSON data with xidel , export as a variable and execute curl with it:

#!/bin/bash
LINODE_API_KEY=1234
domain_id=1931316
domain_name="abx.com"
domain_type="master"
domain_email="hello@abx.com"
domain_ttl=30

if [ -z "${domain_name}" ]; then
   echo "You must provide domain name"
   exit 1
fi

eval "$(
  xidel -s \
  --variable name="$domain_name" \
  --variable type="$domain_type" \
  --variable email="$domain_email" \
  --variable ttl="$domain_ttl" \
  -e '
    json_query_data:=serialize(
      {
        "domain"?:$name,
        "type"?:if ($type="") then () else $type,
        "soa_email"?:if ($email="") then () else $email,
        "ttl_sec"?:if ($ttl="") then () else int($ttl)
      },
      {"method":"json"}
    )
  ' --output-format=bash
)"

curl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${LINODE_API_KEY}" \
-X PUT \
-d "${json_query_data}" \
--url "https://api.linode.com/v4/domains/${domain_id}"

Xidel

Generate and send the JSON data and parse the returning JSON data with xidel (making curl superfluous):

#!/bin/bash
LINODE_API_KEY=1234
domain_id=1931316
domain_name="abx.com"
domain_type="master"
domain_email="hello@abx.com"
domain_ttl=30
  
# This value must be provided first
if [ -z "${domain_name}" ]; then
   echo "You must provide domain name"
   exit 1
fi

xidel -s \
--variable name="$domain_name" \
--variable type="$domain_type" \
--variable email="$domain_email" \
--variable ttl="$domain_ttl" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${LINODE_API_KEY}" \
--method PUT \
-d '{
  serialize(
    {
      "domain"?:$name,
      "type"?:if ($type="") then () else $type,
      "soa_email"?:if ($email="") then () else $email,
      "ttl_sec"?:if ($ttl="") then () else int($ttl)
    },
    {"method":"json"}
  )
}' \
"https://api.linode.com/v4/domains/${domain_id}" \
-e '$json'