Create variable key/value pairs with argparse (python)

Question

I'm using argparse module to set my command line options. I'm also using a dict as a config in my application. Simple key/value store.

What I'm looking for is a possibility to override JSON options using command line arguments, without defining all possible arguments in advance. Something like --conf-key-1 value1 --conf-key-2 value2 , which would create a dict {'key_1': 'value1','key_2': 'value2'} ('-' in the argument is replaced by '_' in the dict). Then I can combine this dict with my JSON config (dict).

So basically I would like to define --conf-* as an argument, where * can be any key and what comes after is the value .

I did find configargparse module, but as far as I can see I start with a dict I already use.

Any ideas how I could approach this?

Answer 1

I had a similar issue and found a very workable pattern that works well with argparse (here three key-pairs: foo, bar and baz:

mycommand par1 --set foo=hello bar="hello world" baz=5

1. Defining the optional, multivalued argument

The set argument must be defined so:

import argparse
parser = argparse.ArgumentParser(description="...")
...
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+',
                        help="Set a number of key-value pairs "
                             "(do not put spaces before or after the = sign). "
                             "If a value contains spaces, you should define "
                             "it with double quotes: "
                             'foo="this is a sentence". Note that '
                             "values are always treated as strings.")
args = parser.parse_args()

The argument is optional and multivalued, with a minimum of one occurrence ( nargs='+' ).

The result is a list of strings eg ["foo=hello", "bar=hello world", "baz=5"] in args.set , which we now need to parse (note how the shell has processed and removed the quotes!).

2. Parsing the result

For this we need 2 helper functions:

def parse_var(s):
    """
    Parse a key, value pair, separated by '='
    That's the reverse of ShellArgs.

    On the command line (argparse) a declaration will typically look like:
        foo=hello
    or
        foo="hello world"
    """
    items = s.split('=')
    key = items[0].strip() # we remove blanks around keys, as is logical
    if len(items) > 1:
        # rejoin the rest:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    """
    Parse a series of key-value pairs and return a dictionary
    """
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

At this point it is very simple:

# parse the key-value pairs
values = parse_vars(args.set)

You now have a dictionary:

values = {'foo':'hello', 'bar':'hello world', 'baz':'5'}

Note how the values are always returned as strings.

This method is also documented as agit gist .

Answer 2

The first thing I'd try is use parse_known_args to handle other arguments, and handle the list of extras with my on routine. Adding the '--conf-' handling to argparse would be more work.

argv = '--conf-key-1 value1 --conf-key-2 value2'.split()
p = argparse.ArgumentParser()
args, extras = p.parse_known_args(argv)

def foo(astr):
    if astr.startswith('--conf-'):
        astr = astr[7:]
    astr = astr.replace('-','_')
    return astr

d = {foo(k):v for k,v in zip(extras[::2],extras[1::2])}
# {'key_1': 'value1', 'key_2': 'value2'}

The extras parsing could be more robust - making sure that there are proper pairs, rejecting badly formed keys, handling = .

Another approach would be to scan sys.argv for --conf- strings, and use those to construct add_argument statements.

keys = [k for k in argv if k.startswith('--conf-')]
p = argparse.ArgumentParser()
for k in keys:
    p.add_argument(k, dest=foo(k))
print vars(p.parse_args(argv))

---

If you would accept '--conf key1 value1 --conf key2 value2 ...' as the input, you could define

parser.add_argument('--conf', nargs=2, action='append')

which would produce:

namespace('conf': [['key1','value1'],['key2','value2']])

which could easily be turned into a dictionary. Or a custom Action could use setattr(namespace, values[0], values[1]) to enter the key/value pairs directly into the namespace.

I believe there have been SO question(s) about accepting '"key1:value" "key2:value2"' inputs.

Answer 3

This can all be done much more simply using str.split(delim, limit) :

class kvdictAppendAction(argparse.Action):
    """
    argparse action to split an argument into KEY=VALUE form
    on the first = and append to a dictionary.
    """
    def __call__(self, parser, args, values, option_string=None):
        assert(len(values) == 1)
        try:
            (k, v) = values[0].split("=", 2)
        except ValueError as ex:
            raise argparse.ArgumentError(self, f"could not parse argument \"{values[0]}\" as k=v format")
        d = getattr(args, self.dest) or {}
        d[k] = v
        setattr(args, self.dest, d)

...


myparser.add_argument("--keyvalue",
                      nargs=1,
                      action=kvdictAppendAction,
                      metavar="KEY=VALUE",
                      help="Add key/value params. May appear multiple times.")

Answer 4

Current code for the top answer has a bug where if the "items" variable in the "parse_var" function doesn't have an "=" it returns the variable "value" before assignment. Here is a modified / condensed function that provides some error handling for whether an "=" was provided in each argument:

def parse_vars(items):
    """
        Parse a series of key-value pairs and return a dictionary and 
        a success boolean for whether each item was successfully parsed.
    """
    count = 0
    d = {}
    for item in items:
        if "=" in item:
            split_string = item.split("=")
            d[split_string[0].strip()] = split_string[1].strip()
            count += 1
        else:
            print(f"Error: Invalid argument provided - {item}")
        
    return d, count == len(items)

Answer 5

To simplify slightly on fralaus answer, the 2 methods can be combined into one easily.

Note: My docstring etc differ as I was using it for ansible extra_vars, but the core logic for string splitting came from fralaus' answer.

 def parse_vars(extra_vars):
     """
     Take a list of comma seperated key value pair strings, seperated
     by comma strings like 'foo=bar' and return as dict.
     :param extra_vars: list[str] ['foo=bar, 'key2=value2']

     :return: dict[str, str] {'foo': 'bar', 'key2': 'value2'}
     """
     vars_list = []
     if extra_vars:
         for i in extra_vars:
            items = i.split('=')
            key = items[0].strip()
            if len(items) > 1:
                value = '='.join(items[1:])
                vars_list.append((key, value))
     return dict(vars_list)

print parse_vars(args.set)

 $ test.py --set blah=gar one=too
>> {"blah": "gar", "one": "too"}

Answer 6

you can use vars() convert the attributes of the 'args' object to dict.

Then print keys and values of the dict.

parser = argparse.ArgumentParser(description="Parameters")
parser.add_argument("--x", type=str, default="")
args, _ = parser.parse_known_args()

for k, v in sorted(vars(args).items()):
    print(k, '=', v)

Answer 7

找到了一个非常简单的解决方案：vars(args) 将其转换为 dict 对象。

Answer 8

here and there partial answers. Let's sum up to the standard argparse way.

import argparse
class kwargs_append_action(argparse.Action):
"""
argparse action to split an argument into KEY=VALUE form
on append to a dictionary.
"""

def __call__(self, parser, args, values, option_string=None):
    try:
        d = dict(map(lambda x: x.split('='),values))
    except ValueError as ex:
        raise argparse.ArgumentError(self, f"Could not parse argument \"{values}\" as k1=v1 k2=v2 ... format")
    setattr(args, self.dest, d)

parser = argparse.ArgumentParser(description="...")
parser.add_argument("-f", "--filters", dest="filters",
                        nargs='*',
                        default={'k1':1, 'k2': "P"},
                        required=False,
                        action=kwargs_append_action,
                        metavar="KEY=VALUE",
                        help="Add key/value params. May appear multiple times. Aggregate in dict")
args = parser.parse_args()

and the usage:

python main.py --filters foo=1 bar="coucou"