shell：参数列表太长

Question

I would like to load test an endpoint with multiple requests at the same time. I am getting the error Argument list too long . How can I fix this?

import subprocess
import argparse
def main(**kwargs):
    my_parser = argparse.ArgumentParser()
    my_parser.add_argument('-s')
    my_parser.add_argument('-e')
    args = my_parser.parse_args()
    i = int(vars(args)['s'])
    cmd = ''
    f= open('test.txt')
    while i <= int(vars(args)['e']):
        for line in f:
            cmd += f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
        i +=1
    subprocess.run(cmd, shell=True)
if __name__ == "__main__":
    main()

$ python load_generator.py -s 1 -e 1
Traceback (most recent call last):
  File "load_generator.py", line 20, in <module>
    main()
  File "load_generator.py", line 17, in main
    subprocess.run(cmd, shell=True)
  File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/bin/sh'

Number of total records

$ cat test.txt | wc -l
18732 

Sample data

$ cat test.txt
'{"ErrorCode": 55201, "ErrorClass": 8, "NodeID": 32, "Params0": 1, "Params1": 70451202, "Params2": 9530, "Params3": 7, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}' 
'{"ErrorCode": 55004, "ErrorClass": 8, "NodeID": 32, "Params0": 538990697, "Params1": 410814, "Params2": 410931, "Params3": 0, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}' 
'{"ErrorCode": 282, "ErrorClass": 0, "NodeID": 32, "Params0": 61, "Params1": 14, "Params2": 0, "Params3": 5, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}'

Answer 1

我最终使用了@barmar 建议，即创建一个.sh文件并通过/bin/sh运行它。

Answer 2

Try change this:

    while i <= int(vars(args)['e']):
        for line in f:
            cmd += f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
        i +=1
    subprocess.run(cmd, shell=True)

to:

    while i <= int(vars(args)['e']):
        for line in f:
            cmd = f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
            subprocess.run(cmd, shell=True)
        i +=1

Or you could check from this " How to start a background process in Python? "