[英]Python asyncio not running asynchronously
In Python 3.10, I am submitting a get request to an endpoint running in an asynchronous manner so that, while a get call is made and I'm waiting for the response, other get requests run asynchronously.在 Python 3.10 中,我向以异步方式运行的端点提交一个获取请求,以便在进行获取调用并且我正在等待响应时,其他获取请求异步运行。 The code is currently running without exception or error but there is no asynchronous activity happening.
该代码当前正在运行,没有异常或错误,但没有发生异步活动。 Instead the code is still running sequentially (one request call at a time from the beginning of the range to the end).
相反,代码仍然按顺序运行(从范围的开始到结束,一次调用一个请求)。 Can anyone spot what I might be doing wrong?
谁能发现我可能做错了什么? Not sure where I've gone wrong.
不知道我哪里出错了。
import json
from src.services import data_transformation_service as dts, connectors_service as cs
import os
import requests as rq
import logging
import sqlalchemy.exc as saexc
import asyncio
import time
# Set globals, get config, set logging level
logging.basicConfig(level=logging.ERROR)
num_movies_to_import = 5
region = 'us-east-1'
config = cs.Config(cs.get_config(os.getenv('config_s3_bucket'), os.getenv('config_s3_file')), region)
# ----------------- Execute API request(s) & Load into Database -----------------
async def get_endpoint_data(endpoint):
return json.loads(rq.get(endpoint).text.replace('\'', ''))
async def retrieve_data(connection, movie):
try:
print(f"{movie}")
endpoint = config.get_movie_db_api_url('movies', movie)
ar = await get_endpoint_data(endpoint)
if sql := dts.sql_insert_creation_movie_api(config.get_movie_table(), ar):
connection.execute(sql) # run insert into database
print(f"Merged Record: {sql}")
else:
print("Skipping record insert.")
except saexc.StatementError:
print(f"Error with record id {movie} most likely missing data or mal structured, skipping to next.")
# except TypeError:
# print("JSON returned by API not valid, skipping to next.")
return True
async def download_movie_data():
with cs.DatabaseConnection('postgres', config.get_config_env(), 'evandb', region) as saq:
results = await asyncio.gather(*[retrieve_data(saq, movie) for movie in range(500, num_movies_to_import + 500)])
print(f'{results}')
# list(range(500, num_movies_to_import + 500))
if __name__ == "__main__":
start = time.time()
asyncio.run(download_movie_data()) # Create the asynch event loop
end = time.time()
print(f"Total Time: {end - start} seconds")
I found my issue... to help others new to asynch - the regular requests library doesn't support asynchronous processing.我发现了我的问题……帮助其他不熟悉异步的人——常规请求库不支持异步处理。 I decided to switch to aiohttp instead.
我决定改用 aiohttp。 For example:
例如:
async def get_endpoint_data(endpoint):
async with aiohttp.ClientSession() as session:
async with session.get(endpoint) as response:
response = await response.json()
return response
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.