Long curl command fails python imports when cleaned up for readability

Question

I have a long curl command that was cleaned up in another answer with backslashes to increase it's readability. The only issue with this is it throws off the small snippet of python code I use inside it. The working one-liner is below: (It talks to an API, I've removed the key variables and anything related to my company.)

NETWORK_ID=$(curl -L -H 'X-Cisco-Meraki-API-Key: mykeygoeshere' -X POST -H'Content-Type: application/json' --data-binary '{"name":"'"$NETWORK_NAME"'", "type":"appliance", "timeZone":"'"$TIME_ZONE"'"}' 'https://dashboard.meraki.com/api/v0/organizations/foobar/networks' | python -c "import sys, json; print json.load(sys.stdin)['id']")

When cleaned up with escape characters, it looks much nicer, increases readability and the ability to edit it. How I'd like it to look is below:

NETWORK_ID=$(curl -L -H 'X-Cisco-Meraki-API-Key: mykeygoeshere'\
    -X POST -H'Content-Type: application/json' --data-binary '{"name":"'"$NETWORK_NAME"'",\
    "type":"appliance", "timeZone":"'"$TIME_ZONE"'"}'\
    'https://dashboard.meraki.com/api/v0/organizations/foobar/networks'\
    | python -c "import sys, json; print json.load(sys.stdin)['id']")

But when I execute the above code, I get thrown these errors:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   285    0   202  100    83    516    212 --:--:-- --:--:-- --:--:--   516
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

What is the functional difference between the one-liner and the readable code? To reiterate, the one-liner completes successfully without issue. The code with the escape characters doe not. How to I get the Python imports to work properly under these conditions?

Answer 1

You are splitting in the middle of a quoted argument (value of --data-binary ), so the spaces on the next line get included in the value.

You want to keep the whole argument on one line, like:

... --data-binary \
   '{"name":"'"$NETWORK_NAME"'", "type":"appliance", "timeZone":"'"$TIME_ZONE"'"}' \
   ...

or put that string in a variable and use that instead:

json='{"name":"'"$NETWORK_NAME"'", "type":"appliance", "timeZone":"'"$TIME_ZONE"'"}'
networkId=$(curl ... --data-binary "$json" \
            ...)

Answer 2

The basic approach is flawed. You should not try to generate JSON using shell parameter interpolation, because the value in the shell parameter is not necessarily a valid JSON value. (Think SQL injection.) Even for hard-coded JSON, it's simpler to read from a here document than to try to quote it properly. In either case, it's best to use a proper tool, like jq or a another Python script, to generate the JSON than to build it by hand.

Finally, define a function to call from inside the command substitution to format everything in a more readable fashion.

get_network_id () {
    local -a curl_options jq_options
    curl_options=(
      -L
      -H 'X-Cisco-Meraki-API-Key: mykeygoeshere'
      -X POST
      -H 'Content-Type: application/json'
      --data-binary @-
      )
    url='https://dashboard.meraki.com/api/v0/organizations/foobar/networks'

    jq_options=( -n --arg netname "$NETWORK_NAME" --arg tz "$TIME_ZONE" )
    jq_filter='
    {
      name: $netname,
      type: "appliance",
      timeZone: $tz
    }'

    jq "${jq_options[@]}" "$jq_filter" |
      curl "${curl_options[@]}" "$url"  |
      jq -r '.id'
}

NETWORK_ID=$(get_network_id)

(You might also make the network name and the timezone arguments to the function instead of global variables.)

Answer 3

You did not provide the output of the curl command, assuming your curl call is correct and printing out a valid json, python2 should read from stdin:

cat /usr/lib/python3.6/site-packages/zmq/utils/config.json | python2 -c 'import sys, json; print(json.loads(sys.stdin.read()))'

returns:

{u'have_sys_un_h': True, u'allow_legacy_libzmq': False, u'zmq_prefix': u'', u'no_libzmq_extension': False, u'build_ext': {}, u'bdist_egg': {}, u'skip_check_zmq': False, u'libzmq_extension': False, u'bundle_msvcp': None}