简体   繁体   English

在 python 中返回多处理输出

[英]Return multiprocessed outputs in python

I am trying to make two functions run in parallel.我试图让两个函数并行运行。 The first functions makes an API call and returns the json output and the second function makes a call to database and returns the data captured.第一个函数进行 API 调用并返回 json output 和第二个 ZC1C425268E68385D1AB5074 调用数据库。

I have the following code -我有以下代码 -

import multiprocessing
ret = {'db': None, 'api':None}

def db_call(queue=None):
    engine = db.create_engine('mysql+pymysql://{}:{}@{}/{}'.format(user, password, host, database))
    dbConnection = engine.connect()
    df_aws_accounts = pd.read_sql(query, dbConnection)
    if queue:
        queue['db'] = df_aws_accounts
    return df_aws_accounts

def api_call(queue=None):
    data = requests.get(url, verify=False)
    df = pd.DataFrame(data)
    if queue:
        queue['api'] = df
    return df

def runInParallel(*fns):
    queue = multiprocessing.Queue()
    queue.put(ret)
    proc = []
    for fn in fns:
        p = Process(target=fn,args=((queue),))
        p.start()
        proc.append(p)
    print(queue.get())
    for p in proc:
        p.join()

l = [api_call, db_call]

runInParallel(l)

when I run the above code -当我运行上面的代码时 -

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
TypeError: 'list' object is not callable

How do I get output from runInParallel and assign that to the variables?如何从 runInParallel 获取 output 并将其分配给变量? Edit- Note - These functions are working individually.编辑-注意-这些功能是单独工作的。 but not when I do it via runInParallel function.但不是当我通过 runInParallel function 执行此操作时。

Edit 2 - Updated the code based on suggestions.编辑 2 - 根据建议更新了代码。

The main problem is that, the function runInParallel is not returning anything主要问题是,function runInParallel没有返回任何东西

But also if you want to make a function which is using multiprocessing return anything, you can create a global variable, like output, and make the function assign value to it like:但是,如果您想创建一个使用多处理返回任何内容的 function,您可以创建一个全局变量,如 output,并让 function 为它赋值:

def api_call():
  global output
  output = api_data

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM