Why does argparse fail when an option is put just before the rest of the positional parameters?

Question

Consider this test argparse example in scratch.py using python 3.7.2:

import sys
import argparse
import yaml


def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--sound', nargs='?', default=None)
    parser.add_argument('greeting')
    parser.add_argument('name')
    parser.add_argument('params', nargs='*')
    return vars(parser.parse_args(sys.argv[1:]))


def main():
    print(yaml.dump((get_args())))


if __name__ == "__main__":
    main()

If I use it:

(websosadmin) ~/wk/cliosoft/websosadmin/sosadmin_cli $ python scratch.py hello john loud red
greeting: hello
name: john
params:
- loud
- red
sound: null

(websosadmin) ~/wk/cliosoft/websosadmin/sosadmin_cli $ python scratch.py --sound bell hello john loud red
greeting: hello
name: john
params:
- loud
- red
sound: bell

(websosadmin) ~/wk/cliosoft/websosadmin/sosadmin_cli $ python scratch.py hello --sound bell john loud red
greeting: hello
name: john
params:
- loud
- red
sound: bell

(websosadmin) ~/wk/cliosoft/websosadmin/sosadmin_cli $ python scratch.py hello john --sound bell loud red
usage: scratch.py [-h] [--sound [SOUND]] greeting name [params [params ...]]
scratch.py: error: unrecognized arguments: loud red

Why does the fourth case fail?

Answer 1

This answer builds on hpaulj 's response.

It is a bug in argparse but there is a workaround: Replace parse_args with parse_intermixed_args .

This works in the example in the question (and in the actual application which is complex with many sub commands and different flags and positional parameters) but it may fail if things like argparse.REMAINDER are used.

Answer 2

With

hello john --sound bell loud red

is parsed as:

greeting = 'hello'
name = 'john'
params = []
sound = 'bell'

['loud','red'] are left over.

The parser alternatively handles positionals and a optional. It uses a regex style of pattern matching to assign the set of positionals upto the next optional ('--sound'). 'greeting' and 'name' each require one string. 'params' can accept any list ('*') including an empty one. Once it's used up there aren't any more arguments left to consume the 'loud' and 'red'.

It's a tricky behavior, and not ideal. There's at least one bug/issue on the topic, but if my memory is correct it's not an easy thing to patch.

Changing to nargs='+' should give the correct parsing.