How to expand environment variable in python while escaping special characters?

Question

On the host, I have an environment variable with a quote sign in the value, something like:

export VALUE_WITH_QUOTE_FROM_OS='quote"value'

and when I echo on bash, it's fine

#echo $VALUE_WITH_QUOTE_FROM_OS
quote"value

I have a following json string:

json_str = '{"key":"${VALUE_WITH_QUOTE_FROM_OS}"}'

Then I want to expand the environment variable inside a python script before further processing, something like this

json_str = os.path.expandvars(json_str)
json_dict = json.loads(json_str)

However, this expansion would break the json syntax, since the json_str had become

'{"key":"quote"value"}' (<== bad unescaped quote in the value)

instead of

'{"key":"quote\"value"}' 

Is there anyway I can inform os.path.expandvars() to escape the double quote when expanding the value? If not, how should I expand the environment variable so that the double quote can be escaped.

Note 1 The value of the environment variable is a security token, so I have to keep the double quote there as it is.

Note 2 Current json interface has already been determined and used extensiely as it is. This json_str is passed to me, therefore I should and must only expand the environment variable denoted by ${} in the json string, no other modification are allowed.

Note 3 This json_str is extremely large with complex, dynamic, nested structure and is consumed by multiple clients who cannot access host os environment variable. Although it's possible for me to load the json_str first, traverse the dictionary to resolve the environment variable, and then dump the dict back to json_str and then dispatch to all the clients, I think it is less efficient compared to processing it just as a string.

Thanks.

Answer 1

I certainly don't guarantee that this will have the portability that you get with os.path , but this should mostly work for systems that use posix I think:

import re
import os
import json

regex = re.compile(r'\$(\w+|\{[^}]*\})')
json_str = '{"key":"${FOO}"}'
def os_expandvar(match):
    v = match.group(1)
    if v.startswith('{') and v.endswith('}'):
        v = v[1:-1]
    return json.dumps(os.environ.get(v, ''))[1:-1]
print(regex.sub(os_expandvar, json_str))

The regular expression (and implementation ideas in general) were borrowed from the implementation of os.path.expandvars in the posixpath module. I removed a lot of the complexity to simplify the answer, but you can put it back in should you find that you need it.

This should handle cases where the substitution strings are $FOO or ${FOO} as is typical on a posix system.

---

There are other options here too ... With the example you gave, you could decode the json first and then expand all of the values. Depending on the format of the json, you might need a recursive function to get the job done:

# untested
def json_expandvars(o):
    if isinstance(o, dict):
       return {json_expandvars(k): json_expandvars(v) for k, v in o.items()}
    elif isinstance(o, list):
       return [json_expandvars(v) for v in o]
    elif isinstance(o, basestring):
       return os.path.expandvars(o)
    else:
       return o

json_dict = json_expandvars(json.loads(json_str))

Answer 2

There's no way for os.path.expandvars to know that the result needs to be a valid JSON string, with quotes escaped.

Instead, expand the environment variable after you decode the JSON into a dictionary.

json_dict = json.loads(json_str);
json_dict['key'] = os.path.expandvars(json_dict['key']);