Python多重处理：子进程以不同的速度运行

Question

I'm new to python multiprocessing . I'm trying to use a third-party web-API to fetch data for multiple symbols of interest. Here is my python code:

<!-- language:lang-py-->

def my_worker(symbol, table_name):
    while True:
        # Real-time data for the symbol, third party code which is verified
        data = webApi.getData(symbol)
        query = ('insert into ' + table_name + '(var1, var2) values("%s, %s")' %(data[0], data[1]))
        # Execute query and store the data. Omitted for sake of brevity

if __name__ == "__main__":
    my_symbols = get_symbols_list() # List of symbols
    my_tables = get_tables_list()   # Corresponding list of mysql tables
    jobs = []
    for pidx in range(len(my_symbols)):
        pname = 'datarecorder_' + my_symbols[pidx]  # Naming the process for later identification
        p = multiprocessing.Process(name=pname, target=my_worker, args=(my_symbols[pidx], my_tables[pidx],))
        jobs.append(p)
        p.start()

There are approximately 50 processes created in this code.

Problem that I'm facing: is that when I look into the corresponding tables after a certain amount of time (say 5 minutes), the number of records in each of the table in my_tables is drastically different (on the order of multiple of 10s)

Since I am using the same API, the same network connection and the same code to fetch and write data to the mysql tables, I'm not sure what is causing this difference in number of records. My hunch is that each of the 50 processes is getting assigned an unequal amount of RAM and other resources, and perhaps the priority is also different(?)

Can someone tell me how can I ensure that each of these processes poll the webApi roughly equal number of times?

Answer 1

An effective way to approach such things is to start with something vastly simpler, then add stuff to it until "a problem" shows up. Otherwise it's just blind guesswork.

For example, here's something much simpler, which I'm running under Windows (like you - I'm using a current Win10 Pro) and Python 3.5.2:

import multiprocessing as mp
from time import sleep

NPROCS = 50

def worker(i, d):
    while True:
        d[i] += 1
        sleep(1)

if __name__ == "__main__":
    d = mp.Manager().dict()
    for i in range(NPROCS):
        d[i] = 0

    ps = []
    for i in range(NPROCS):
        p = mp.Process(target=worker, args=(i, d))
        ps.append(p)
        p.start()

    while True:
        sleep(3)
        print(d.values())

Here's the most recent output after about a minute of running:

[67, 67, 67, 67, 67, 67, 67, 67, 67, 67,
 67, 67, 67, 67, 67, 67, 67, 67, 67, 66,
 66, 66, 66, 66, 66, 66, 66, 66, 66, 66,
 66, 66, 66, 66, 66, 66, 66, 66, 66, 66,
 66, 66, 66, 66, 66, 66, 66, 66, 66, 66]

So I can conclude that there's nothing "inherently unfair" about process scheduling on this box. On your box? Run it and see ;-)

I can also see in Task Manager that all 50 processes are treated similarly, with (for example) the same RAM usage and priority. FYI, this box happens to have 8 logical cores (4 physical), and way more than enough RAM (16GB).

Now there are worlds of additional complications in what you're doing, none of which we can guess from here. For example, maybe you're running out of RAM and so some processes are greatly delayed by pagefile swapping. Or maybe the work you're doing takes much longer for some arguments than others. Or ... but, regardless, the simplest way to find out is to incrementally make a very simple program a little fancier at a time.