简体   繁体   中英

Return multiprocessed outputs in python

I am trying to make two functions run in parallel. The first functions makes an API call and returns the json output and the second function makes a call to database and returns the data captured.

I have the following code -

import multiprocessing
ret = {'db': None, 'api':None}

def db_call(queue=None):
    engine = db.create_engine('mysql+pymysql://{}:{}@{}/{}'.format(user, password, host, database))
    dbConnection = engine.connect()
    df_aws_accounts = pd.read_sql(query, dbConnection)
    if queue:
        queue['db'] = df_aws_accounts
    return df_aws_accounts

def api_call(queue=None):
    data = requests.get(url, verify=False)
    df = pd.DataFrame(data)
    if queue:
        queue['api'] = df
    return df

def runInParallel(*fns):
    queue = multiprocessing.Queue()
    queue.put(ret)
    proc = []
    for fn in fns:
        p = Process(target=fn,args=((queue),))
        p.start()
        proc.append(p)
    print(queue.get())
    for p in proc:
        p.join()

l = [api_call, db_call]

runInParallel(l)

when I run the above code -

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
TypeError: 'list' object is not callable

How do I get output from runInParallel and assign that to the variables? Edit- Note - These functions are working individually. but not when I do it via runInParallel function.

Edit 2 - Updated the code based on suggestions.

The main problem is that, the function runInParallel is not returning anything

But also if you want to make a function which is using multiprocessing return anything, you can create a global variable, like output, and make the function assign value to it like:

def api_call():
  global output
  output = api_data

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM