argparse solution requested for commands and subcommands with subcommands

Question

Hopefully this will translate into an elegant solution, but i am unable to figure it out myself. I have been reading huge amounts of examples and explanations but I cant seem to get it to work.

I am writing a program that needs the following options:

myprogram:
-h              help
--DEBUG         debugging on

Commands are: dashboard / promote / deploy / auto-tag / dbquery
Some commands have sub-commands.

Dashboard prosesing:
  dashboard <ALL | DEBUG | AAA [sub1-aaa |  ...] >

Promoting:
  promote <environment> [to environment] <system> [version]

Deployment
  deploy <system> [version] <in environment>

Auto-tagging
  auto-tag <repo> <project> [--days-back]

Database queries  (--tab is optional)
         (system or env or both are required)
  dbquery lock set    <system | environment>   [--tab]
  dbquery lock clear  <system | environment>   [--tab]
  dbquery lock show   <system | environment | before-date | after-date>   [--tab]

  dbquery promote list <system| version| environment | before-date | after-date>  [--tab]
  dbquery deploy list <system| version| environment | before-date | after-date>   [--tab]

If a command is used, some sub-commands or options are required.

I have a hard time getting this done using the argparse library. I tried to use add_argument_group, subparsers etc. But I think I am missing something basic here. All examples i found getting close are about svn, but they seem to only go 1 level after svn. And i need more, or a different approach.

If possible I would like to make all parameters after dbquery deploy list optional, with at least 1 option required. But to distinguish between the name of a system and an environemnt might become tricky so it might be better to change this:

dbquery lock set    <system | environment>

into

dbquery lock set    <system=system | env=environment>

ps Options enclosed between [] are optional, options between <> are required.

Thanks in advance.

In response to the comment of providing my code, lets focus on the dbquery, because the rest might be a repetition:

import argparse
parser = argparse.ArgumentParser(description="Main cli tool for processing in the CD pipeline (%s)" % VERSION)
subparsers = parser.add_subparsers(help='additional help',title='subcommands')
dbq_parser=subparsers.add_parser("dbqueue", aliases=['dbq'])
dbq_group_lock = dbq_parser.add_argument_group('lock', 'lock desc')
dbq_group_promote =dbq_parser.add_argument_group('promote')
dbq_group_deploy = dbq_parser.add_argument_group('deploy','Deployment description')

dbq_group_lock.add_argument('set', help="Sets lock")
dbq_group_lock.add_argument('clear', help='Clears lock')
dbq_group_lock.add_argument('show', help='Show lock status')

dbq_group_deploy.add_argument('name system etc')

Executing results in:

# python3 main.py -h
usage: main.py [-h] [--debug] {dbqueue,dbq} ...

Main cli tool for processing in the CD pipeline (cdv3, Jun 2019, F.IJskes)

optional arguments:
  -h, --help     show this help message and exit
  --debug        Generate debug output and keep temp directories

subcommands:
  {dbqueue,dbq}  additional help

This looks okay, but:

#python3 main.py dbq -h
usage: main.py dbqueue [-h] set clear show name system etc

optional arguments:
  -h, --help       show this help message and exit

lock:
  lock desc

  set              Sets lock
  clear            Clears lock
  show             Show lock status

shows that the expected parameters are not lock, promote or deploy.

Okay, the feedback helped my understanding. I now understand that parsers can get subparsers and those can get parsers. So there might be no limit in the depth one can go. This new insight got me to this: (a partial copy from my working example)

import argparse
main_parser = argparse.ArgumentParser(description="Main cli tool for processing in the CD pipeline (%s)" % VERSION)

main_subparsers = main_parser.add_subparsers(help='',title='Possible commnds')

dashbrd_subparser = main_subparsers.add_parser('dashboard', help="Proces specified dashboards", allow_abbrev=True)
dashbrd_subparser.add_argument('who?',help='Proces dashboard for ALL, Supplier or DEBUG what should happen.')
dashbrd_subparser.add_argument('-subsystem', help='Select subsystem to proces for given supplier')

dbq_main=main_subparsers.add_parser("dbquery", help="Query database for locks,deployments or promotes")
dbq_main_sub=dbq_main.add_subparsers(help="additions help", title='sub commands for dbquery')
dbq_lock=dbq_main_sub.add_parser('lock', help='query db for lock(s)')
dbq_lock_sub=dbq_lock.add_subparsers(help='', title='subcommands for lock')
dbq_lock_sub_set=dbq_lock_sub.add_parser('set', help='sets a lock')
dbq_lock_sub_set.add_argument('-env', required=True)
dbq_lock_sub_set.add_argument('--tab',required=False, action="store_true" )
dbq_lock_sub_clear=dbq_lock_sub.add_parser('clear', help='clears a lock')
# dbq_lock_sub_set.add_argument('-env', required=True)
# dbq_lock_sub_set.add_argument('--tab', required=False)
dbq_lock_sub_show=dbq_lock_sub.add_parser('show', help='shows a lock/locks')
# dbq_lock_sub_set.add_argument('-env', required=True)
# dbq_lock_sub_set.add_argument('--tab', required=False)

print( vars(main_parser.parse_args()))
exit(1)

I now only seem to have issues using parameters as '-env' and '-subsystem' across the different sub-commands. Because there is a conflict when i add them to another parser. I also have no data to use about what options where chosen. This is something that is also needed.

Answer 1

Almost there, and what I now have is usable to a great extent, so I'll post this so other users might benefit from my work, which has benefited by the comments of hpaulj which led me to other documents that could clarify other parts.

# see: https://pymotw.com/3/argparse/

# BLOCK FOR SHARED options used at more places,
# see also: https://stackoverflow.com/questions/7498595/python-argparse-add-argument-to-multiple-subparsers
env_shared = argparse.ArgumentParser(add_help=False)
env_shared.add_argument('-env', action="store", help='name of environment <tst|acc|acc2|prod|...>', metavar='<environment>', required=True)
ds_shared = argparse.ArgumentParser(add_help=False)
ds_shared.add_argument('-ds', action="store", help='name of subsystem to use <ivs-gui|ivs-vpo|...>', metavar='<system name>', required=True)
version_shared = argparse.ArgumentParser(add_help=False)
version_shared.add_argument('-version', action="store", help='version to use. If used, specify at least sprint as in 1.61', metavar='<version>')
tab_shared=argparse.ArgumentParser(add_help=False)
tab_shared.add_argument('--tab', required=False, action="store_true", help='Use nice tabulation for output')
# END OF SHARED options

# MAIN
main_p = argparse.ArgumentParser(description="Main cli tool for use in the CD v3 pipeline (%s)" % VERSION, epilog='Defaults are taken from the configuration file, which is pulled from git when needed.', )
# Change formatting of help output for main-parser
main_p.formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=30, width=80)
main_p.add_argument("--debug", action="store_true", help="Generate debug output and keep temp directories")

# MAIN-SUB
main_s_p = main_p.add_subparsers(title='Commands', dest='main_cmd')

# MAIN-SUB-DASHBOARD
dashbrd_p = main_s_p.add_parser('dashboard', aliases=['db'], help="Proces specified dashboards", parents=[ds_shared])
dashbrd_p.formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=40, width=80)
dashbrd_p.add_argument('which', help='Proces dashboard for <ALL|Supplier|DEBUG what would happen>. If supplier is specified an optional subsystem to be processed can be passed using -ds')

# MAIN-SUB-QUERY
query_p = main_s_p.add_parser("query", aliases=['qry'], help="Query database for locks,deployments or promotes")
query_p.formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=40, width=80)
# MAIN-SUB-QUERY-SUB
query_s_p = query_p.add_subparsers(help="additions help", title='sub commands for query', dest='query_cmd')
# MAIN-SUB-QUERY-LOCK
q_lock_p = query_s_p.add_parser('lock', help='query db for lock(s)', )
# MAIN-SUB-QUERY-LOCK-SUB
q_lock_sub = q_lock_p.add_subparsers(help='', title='subcommands for dbquery lock', dest='lock_cmd')
# MAIN-SUB-QUERY-LOCK-SUB-SET
q_lock_set_p = q_lock_sub.add_parser('set', help='sets a lock', parents=[env_shared, ds_shared])
# MAIN-SUB-QUERY-LOCK-SUB-CLEAR
q_lock_clear_p = q_lock_sub.add_parser('clear', help='clears a lock', parents=[env_shared, ds_shared], )
# MAIN-SUB-QUERY-LOCK-SUB-SHOW
q_lock_show_p = q_lock_sub.add_parser('show', help='shows a lock/locks', parents=[env_shared, tab_shared])


# MAIN-SUB-QUERY-PROMOTE
q_promote_p = query_s_p.add_parser('promote', help='query db for promotions', parents=[env_shared,ds_shared] )

# MAIN-SUB-QUERY-DEPLOY
q_deploy_p = query_s_p.add_parser('deploy', help='query db for deployments')

# MAIN-SUB-PROMOTE
promote_p = main_s_p.add_parser('promote', help='Do a promotion.', parents=[env_shared, ds_shared, version_shared])
promote_p.add_argument('-dest', required=False, action="store", help='If used specifies the destination, otherwise defaults are used. (-dest is optional, -env is not)', metavar='<dest_env>')

# MAIN-SUB-DEPLOY
deploy_p = main_s_p.add_parser('deploy', help='Deploy software', parents=[ds_shared, env_shared])

# MAIN-SUB-AUTOTAG
autotag_p = main_s_p.add_parser('autotag', help="Autotags specified repository", parents=[ds_shared])

print("------------arguments-----------")
print(vars(main_p.parse_args()))

This creates a lot of the functionality i intended.