How to pass a function, list, among others as args to argparse in Python?

Question

I have the following script which works, and I'm trying to avoid using '__main__' inside the same module:

def download():
    urls = \
        [
            'https://ipleak.net/json',
            'https://httpbin.org/get'
        ] * 4

    downloads = asyn.init_download(urls, "json")
    return downloads

def pprint_json(d):
    print(json.dumps(d, indent=4, sort_keys=True))

def multiprocess_list(n_pools, func, list):
    executor = concurrent.futures.ProcessPoolExecutor(n_pools)
    futures = [executor.submit(func, item) for item in list]
    concurrent.futures.wait(futures)

if __name__ == '__main__':
    multiprocess_list(4, pprint_json, download())

The download() function, as its name depicts, downloads the urls asynchronously using asyncio and aiohttp .

I want to execute multiprocess_list from anywhere 'globally':

def multiprocess_list(n_pools, func, list):
    executor = concurrent.futures.ProcessPoolExecutor(n_pools)
    futures = [executor.submit(func, item) for item in list]
    concurrent.futures.wait(futures)

def main(args):
    parser = argparse.ArgumentParser(description="Multiprocessing a list.")
    parser.add_argument("-n", "--n_pools", type=int, required=True)
    parser.add_argument("-f", "--function", required=True)
    parser.add_argument("-l", "--list", required=True)
    args = parser.parse_args(args)

    multiprocess_list(args.n_pools, args.function, args.list)

if __name__ == '__main__':
    import sys
    main(sys.argv[1:])

Import the above module into any other python file, and perhaps run it like: (doesn't work though)

def download():
    urls = \
        [
            'https://ipleak.net/json',
            'https://httpbin.org/get'
        ] * 4

    downloads = asyn.init_download(urls, "json")
    return downloads

def pprint_json(d):
    print(json.dumps(d, indent=4, sort_keys=True))

mp.main(["-n", 4, "-f", pprint_json, "-l", download()])

This gives me an error:

if not arg_string[0] in self.prefix_chars: TypeError: 'int' object is not subscriptable

Hence, pass one argument as a function to be run, another as a list or a function which returns a list like download() .

• Can this be done in python?
• If yes, can someone please explain how?
• Is my approach correct or am I completely losing it?

NB : My interpreter uses Python3.8 and I'm kind of new to python, please bear with me a little.

Answer 1

Argparse expects a list of strings and may choke on other types. Does it work if you quote the 4 ?

mp.main(["-n", "4", "-f", pprint_json, "-l", download()])

pprint_json and the result of download() should also be strings for this to work.

This approach of creating a substitute argv is not always crazy, but in your case, why call main() to parse the args for you if you've already have them? Why not call multiprocess_list() directly with appropriate arguments?

The reason for using main() is because I might add more functions in the future apart from multiprocess_list()

Then you can call those directly when invoking the script from Python, instead of creating another argument to select one of them. You can still use main() to parse the args from the command line.

next error TypeError: 'function' object is not subscriptable

Ah, that wasn't a string either. In that case I don't expect it to work from the command line either. Did you get that much working yet?

You could try something like

mp.main(["-n", "4", "-f", "pprint_json", "-l", download()])

But then main() has to be able to interpret that function name as a function somehow. Maybe something like

multiprocess_list(args.n_pools, getattr(foo, args.function), args.list)

where foo is the module where you keep the selectable functions.