Argparse outputting help text twice

Question

After an hour googling, I can't find anybody who has had anything resembling this issue besides myself. I created a command line interface with argparse. Originally I had tried to leverage argparse's built in help text behavior. But my boss isn't satisfied with the default help text, so he is having me write up the full usage/help text in a text file and just display the entire file.

For some reason, in a certain case, its outputting the text twice.

Here is the basics of how my program is broken down:

I have a top level parser. I read in my help text file, set it to a string help_text, and then set "usage=help_text" on the parser. Then I create subparsers (4 of them and then a base case) to create subcommands. Only one of those subparsers has any additional arguments (one positional, one optional). Before I reworked the help text, I had help text for each individual subcommand by using "help=" but now those are all blank. Lastly, I have set up a base case to display the help text whenever no subcommands are given.

Here is the behavior I'm getting:

When I call the main function with no subcommands and no arguments, my help_text from the text file outputs, and then like 2-3 additional lines of boiler plate I can't seem to get rid of. Also because the word usage appears in my text file, it says "usage: usage"

When I call the main command and then type --help, the exact same thing happens as above.

When I call the one subcommand that has a required positional argument and I don't include that argument... it spits out the entire help text twice. Right above the second time it prints, it prints the default usage line for that subcommand.

Lastly, when I use a different subcommand that has no arguments and give it an argument (one too many) it spits out everything completely correctly without even the extra couple lines at the end.

I don't know how to make heads or tales about this. Here is the main function of the script (I can verify that this problem occurs only in the main function where argparse is used, not the other functions that the main function calls):

def main():
    # Import help text from file
    p = Path(__file__).with_name("help_text.txt")
    with p.open() as file:
        help_text = file.read()

    # Configure the top level Parser
    parser = argparse.ArgumentParser(prog='hubmap-clt', description='Name of cli', usage=help_text)
    subparsers = parser.add_subparsers()

    # Create Subparsers to give subcommands
    parser_transfer = subparsers.add_parser('subcommandone')
    parser_transfer.add_argument('argument1', type=str)
    parser_transfer.add_argument('--optionalargument', default='mydefault')
    parser_login = subparsers.add_parser('subcommandtwo')
    parser_whoami = subparsers.add_parser('subcommandthree')
    parser_logout = subparsers.add_parser('subcommandfour')

    # Assign subparsers to their respective functions
    parser_subcommandone.set_defaults(func=subcommandone)
    parser_subcommandtwo.set_defaults(func=subcommandtwo)
    parser_subcommandthree.set_defaults(func=subcommandthree)
    parser_subcommandfour.set_defaults(func=subcommandfour)
    parser.set_defaults(func=base_case)

    # Parse the arguments and call appropriate functions
    args = parser.parse_args()
    if len(sys.argv) == 1:
        args.func(args, parser)
    else:
        args.func(args)

So to clarify:

Why does the extra couple lines of boiler-plat help text appear sometimes which looks like this:

 name of cli

 positional arguments:
     {subcommandone,subcommandtwo,subcommandthree,subcommandfour}

 optional arguments:
    -h, --help            show this help message and exit

Why does using subcommandone with too few arguments print out the help text twice (but NOT the extra lines of boiler-plate help text.

why does using subcommandtwo with one too MANY arguments print everything perfectly without any extra lines?

Answer 1

With a modification of your main :

def foo():
    # Import help text from file
    # p = Path(__file__).with_name("help_text.txt")
    # with p.open() as file:
    #    help_text = file.read()
    help_text = "cli usage: foobar\n morebar"

    # Configure the top level Parser
    parser = argparse.ArgumentParser(
        prog="hubmap-clt", description="Name of cli", usage=help_text
    )
    subparsers = parser.add_subparsers()

    # Create Subparsers to give subcommands
    parser_transfer = subparsers.add_parser("subcommandone")
    parser_transfer.add_argument("argument1", type=str)
    parser_transfer.add_argument("--optionalargument", default="mydefault")
    parser_login = subparsers.add_parser("subcommandtwo")
    # parser_whoami = subparsers.add_parser("subcommandthree")
    # parser_logout = subparsers.add_parser("subcommandfour")

    # Assign subparsers to their respective functions
    parser_transfer.set_defaults(func="subcommandone")
    parser_login.set_defaults(func="subcommandtwo")
    # parser_subcommandthree.set_defaults(func="subcommandthree")
    # parser_subcommandfour.set_defaults(func="subcommandfour")
    parser.set_defaults(func="base_case")

    return parser

in an iteractive ipython session:

In [8]: p = foo()

In [9]: p.print_usage()
usage: cli usage: foobar
 morebar

Usage is exactly as I specified. And the help for the main parser:

In [10]: p.print_help()
usage: cli usage: foobar
 morebar

Name of cli

positional arguments:
  {subcommandone,subcommandtwo}

optional arguments:
  -h, --help            show this help message and exit

That's what I expect given the arguments.

Help for a subparser:

In [11]: p.parse_args(["subcommandone", "-h"])
usage: cli usage: foobar
 morebar subcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1

positional arguments:
  argument1

optional arguments:
  -h, --help            show this help message and exit
  --optionalargument OPTIONALARGUMENT

Usage is like the main's but with some added info on how to call this subparser and its arguments.

Error when calling the subparsers without enough values:

In [15]: p.parse_args(["subcommandone"])
usage: cli usage: foobar
 morebar subcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
cli usage: foobar
 morebar subcommandone: error: the following arguments are required: argument1

Is this repeat of cli usage that bothering you? This error is raised by the subparser, and I suspect the extra comes from the prog of that subparser. I think I saw something like this on the Python bug/issues for argparse .

error with too much:

In [17]: p.parse_args(["subcommandone", "test", "extra"])
usage: cli usage: foobar
 morebar
hubmap-clt: error: unrecognized arguments: extra

In this case error is produced by the main parser, hence the "hubmat-clt" prog .

change prog :

...: parser_transfer = subparsers.add_parser(...: "subcommandone", prog="hubmap-clt sobcommandone"...: )

In [21]: p.parse_args(["subcommandone", "test", "extra"])
usage: cli usage: foobar
 morebar
hubmap-clt: error: unrecognized arguments: extra

In [22]: p.parse_args(["subcommandone"])
usage: hubmap-clt sobcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
hubmap-clt sobcommandone: error: the following arguments are required: argument1

[21] is as before [17]. But [22] is now showing the prog that I set. I could also have specified a custom usage for the subparser.

If I modify the function to use default usage and prog, but also display the subparser's prog. And I gave the main an "main_foo" positional argument:

In [30]: p = foo()
hubmap-clt main_foo subcommandone
In [31]: p.parse_args(["subcommandone"])
Out[31]: Namespace(main_foo='subcommandone')
In [32]: p.parse_args(["foo", "subcommandone"])
usage: hubmap-clt main_foo subcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
hubmap-clt main_foo subcommandone: error: the following arguments are required: argument1

Notice how the main's usage has been incorporated into the 'prog' for the subparser.

In the bug/issue I found the main parser's usage gets incorporated into the prog of the subparser. That's why you see the duplicate.

https://bugs.python.org/issue42297 [argparse] Bad error message formatting when using custom usage text

The relatively recent date of this bug issue indicates that custom usage is not that common, and even less so when used with subparsers. As my post on this issue indicates, the relation between the main parser, the "subparsers" command, and individual subparsers gets complicated.