[英]How can I send an HTTP request from my FastAPI app to another site (API)?
I am trying to send 100 requests at a time to a server http://httpbin.org/uuid
using the following code snippet我正在尝试使用以下代码片段一次向服务器
http://httpbin.org/uuid
发送 100 个请求
from fastapi import FastAPI
from time import sleep
from time import time
import requests
import asyncio
app = FastAPI()
URL= "http://httpbin.org/uuid"
# @app.get("/")
async def main():
r = requests.get(URL)
# print(r.text)
return r.text
async def task():
tasks = [main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main(),main()]
# print(tasks)
# input("stop")
result = await asyncio.gather(*tasks)
print (result)
@app.get('/')
def f():
start = time()
asyncio.run(task())
print("time: ",time()-start)
I am using FastAPI with Asyncio to achieve the lowest time possible around 3 seconds or less but using the above method I am getting an overall time of 66 seconds that is more than a minute.我正在使用带有 Asyncio 的 FastAPI 来实现大约 3 秒或更短的最短时间,但使用上述方法,我得到的总时间为 66 秒,超过一分钟。 I also want to keep the
main
function for additional operations on r.text
.我还想保留
main
函数以在r.text
上进行其他操作。 I understand that to achieve such low time, concurrency is required but I am not sure what mistake I'm doing here.我知道要达到如此短的时间,需要并发,但我不确定我在这里犯了什么错误。
requests
is a synchronous library. requests
是一个同步库。 You need to use an asyncio
-based library to make requests asynchronously.您需要使用基于
asyncio
的库来异步发出请求。
httpx
is typically used in FastAPI applications to request external services. httpx
通常在 FastAPI 应用程序中用于请求外部服务。 It provides synchronous and asynchronous clients which can be used in def
and async def
path operations appropriately.它提供了同步和异步客户端,可以适当地用于
def
和async def
路径操作。 It is also recommended for asynchronous tests of application.也推荐用于应用程序的异步测试。 I would advice using it by default.
我建议默认使用它。
from fastapi import FastAPI
from time import time
import httpx
import asyncio
app = FastAPI()
URL = "http://httpbin.org/uuid"
async def request(client):
response = await client.get(URL)
return response.text
async def task():
async with httpx.AsyncClient() as client:
tasks = [request(client) for i in range(100)]
result = await asyncio.gather(*tasks)
print(result)
@app.get('/')
async def f():
start = time()
await task()
print("time: ", time() - start)
Output输出
['{\n "uuid": "65c454bf-9b12-4ba8-98e1-de636bffeed3"\n}\n', '{\n "uuid": "03a48e56-2a44-48e3-bd43-a0b605bef359"\n}\n',...
time: 0.5911855697631836
aiohttp
can also be used in FastAPI applications, if you prefer one.如果您愿意,
aiohttp
也可以在 FastAPI 应用程序中使用。
from fastapi import FastAPI
from time import time
import aiohttp
import asyncio
app = FastAPI()
URL = "http://httpbin.org/uuid"
async def request(session):
async with session.get(URL) as response:
return await response.text()
async def task():
async with aiohttp.ClientSession() as session:
tasks = [request(session) for i in range(100)]
result = await asyncio.gather(*tasks)
print(result)
@app.get('/')
async def f():
start = time()
await task()
print("time: ", time() - start)
If you want to limit the number of requests executing in parallel, you can use asyncio.semaphore
like so:如果要限制并行执行的请求数,可以像这样使用
asyncio.semaphore
:
MAX_IN_PARALLEL = 10
limit_sem = asyncio.Semaphore(MAX_IN_PARALLEL)
async def request(client):
async with limit_sem:
response = await client.get(URL)
return response.text
@Alex Noname has made a good point of using asynchronous request library. @Alex Noname 很好地使用了异步请求库。 If you want to make code faster I would suggest using asyncio.Queue as an alternate.
如果您想让代码更快,我建议您使用asyncio.Queue作为替代。 In this example I spun up 100 producers and 100 consumers.
在这个例子中,我创建了 100 个生产者和 100 个消费者。 you can limit the maximum number of messages in the queue like, then producer waits until there is space for new message
您可以限制队列中的最大消息数,例如,生产者等待直到有空间容纳新消息
asyncio.Queue(maxsize=100)
asyncio.Queue(maxsize=100)
also I have made use of AsyncClient from httpx .我也使用了来自httpx的AsyncClient 。
If you want to know more about queues I would suggest this article https://realpython.com/async-io-python/如果您想了解有关队列的更多信息,我建议您阅读这篇文章https://realpython.com/async-io-python/
from time import time
from typing import List
from fastapi import FastAPI
from httpx import AsyncClient
app = FastAPI()
URL = "http://httpbin.org/uuid"
client = AsyncClient()
async def main():
r = await client.get(URL)
return r.text
async def producer(queue: asyncio.Queue):
await queue.put(main)
async def consumer(queue: asyncio.Queue, resp: List):
# await queue.get() == main -> without arguments
resp.append(await (await queue.get())())
async def task():
q = asyncio.Queue(maxsize=100)
response = []
consumers = []
producers = []
[consumers.append(consumer(q, response)) for c in range(100)]
[producers.append(producer(q)) for p in range(100)]
await asyncio.gather(*producers)
await asyncio.gather(*consumers)
print(response)
@app.get('/')
def f():
start = time()
asyncio.run(task())
print("time: ", time() - start)
if __name__ == '__main__':
f()
output输出
['{\n "uuid": "a7713d07-ea5d-40d3-95b4-6673f3c50a8b"\n}\n', '{\n "uuid": "c93f8b89-2c44-40fa-9e5f-736e22ad5f23"\n}\n', '{\n "uuid": "cbb4ad76-7790-45ae-87f1-e425eddc8021"\n}\n', '{\n "uuid": "4c1d81c0-ae7d-401a-99df-e98af3651335"\n}\n', '{\n "uuid": "c5f70738-fbba-4cf9-8fdf-29f8b4eabe63"\n}\n', '{\n "uuid": "d016b852-4312-4502-a336-a6a110237d1d"\n}\n', '{\n "uuid": "22d8b00b-4266-4236-b5a3-ed5d7c5be416"\n}\n', '{\n "uuid": "cd54fdbb-7de9-4df3-90cc-e6b108d5fdf8"\n}\n', '{\n "uuid": "757f0a26-7896-4a04-bea2-60c66a38b05b"\n}\n', '{\n "uuid": "72eb6584-21f4-449b-b6bd-d0f88666126f"\n}\n', '{\n "uuid": "b3deadf5-5b79-491b-829c-0404c306cb68"\n}\n', '{\n "uuid": "789e7422-493d-49d2-9585-e5ca34b7cf36"\n}\n', '{\n "uuid": "48d29a82-ff7c-41f5-8af2-42784326a31f"\n}\n', '{\n "uuid": "84b2d67c-331c-4037-b6e4-c299d93c1899"\n}\n', '{\n "uuid": "386e79f9-073a-4f27-961c-7befcdf95cd4"\n}\n', '{\n "uuid": "8dfdb5e4-dd69-4043-b174-48ec8505f36f"\n}\n', '{\n "uuid": "633e634b-b107-42bb-a7d3-c6bbfff089a0"\n}\n', '{\n "uuid": "962d665f-8663-4be7-a3c6-9426ba500bf4"\n}\n', '{\n "uuid": "320fb858-a751-4c34-9cdb-ddd2f4e28efa"\n}\n', '{\n "uuid": "46a75693-5255-4ac7-8d7a-54910b4d6f68"\n}\n', '{\n "uuid": "5323734b-7ff9-455e-ba5a-66383e6b9a1f"\n}\n', '{\n "uuid": "622a579f-35b6-4e4b-9dba-a8a69c2049c8"\n}\n', '{\n "uuid": "593d5e82-cef3-4be0-99ab-e3034855d7a1"\n}\n', '{\n "uuid": "80f139df-2a27-40c1-8329-e4faa035c45c"\n}\n', '{\n "uuid": "a97e084c-4d30-4c7b-a96e-89ed00dcfe2a"\n}\n', '{\n "uuid": "360d49eb-7222-4064-81c2-6eba2d43a9a5"\n}\n', '{\n "uuid": "a81b6eab-a646-4e58-b986-96a90baa52aa"\n}\n', '{\n "uuid": "0160337e-b400-41d6-ae89-aa46c5131f40"\n}\n', '{\n "uuid": "e600722f-8c15-4959-948b-4c4e5296feb2"\n}\n', '{\n "uuid": "f15403e4-3674-43b2-a0c9-649fd828ba7e"\n}\n', '{\n "uuid": "36bf139c-cc18-45a8-bc55-e7f90ce290b5"\n}\n', '{\n "uuid": "b2368a3c-d86b-4fcd-a0d3-bf7f8f657a83"\n}\n', '{\n "uuid": "d9f16c36-3572-4c70-8a41-3d4e279d76bf"\n}\n', '{\n "uuid": "796087cc-a202-40dd-9921-14802a73323d"\n}\n', '{\n "uuid": "089fa0d7-4c48-4daa-a80d-cb5ebd37dfb7"\n}\n', '{\n "uuid": "e5582bc7-0f8a-4da7-b640-79a0d812154d"\n}\n', '{\n "uuid": "bac0640b-0d0b-4bf2-a3c1-36bdda7cce03"\n}\n', '{\n "uuid": "b4353004-02b2-4846-8692-33dd77ad1d3f"\n}\n', '{\n "uuid": "1b34a744-d0ea-4acf-8bda-33743800d86a"\n}\n', '{\n "uuid": "4d9dd269-6ee2-4356-9bc4-ddf188445320"\n}\n', '{\n "uuid": "a1f380df-0c0d-4aee-bbb7-c3e99fbfe54f"\n}\n', '{\n "uuid": "7cb762eb-1a42-433d-97ea-aa9de4504e35"\n}\n', '{\n "uuid": "981c40e2-64bf-4746-8103-9430bda2a5ca"\n}\n', '{\n "uuid": "22b778eb-82d1-48b9-9874-5ebb80ddb8b1"\n}\n', '{\n "uuid": "e7a9e0e8-7964-400c-aafe-9c36b9b7e1a0"\n}\n', '{\n "uuid": "21a59b91-2732-4bb6-a47e-84008a03c20c"\n}\n', '{\n "uuid": "a78eeb39-5ecb-4509-87c2-b4a2529e3536"\n}\n', '{\n "uuid": "4a332579-ce03-4f69-9db5-78da9196d6b2"\n}\n', '{\n "uuid": "55fbc34f-4eb3-4356-98e3-1df38054a4b2"\n}\n', '{\n "uuid": "257ac454-09c2-4fd4-bdb3-303495360fa2"\n}\n', '{\n "uuid": "7505cc0d-01b3-47f8-91d4-3e54d0f387de"\n}\n', '{\n "uuid": "0fd67af2-622e-4688-b3c8-f64e20f1f3ec"\n}\n', '{\n "uuid": "07653ccf-f408-4807-8ff5-e6098d657451"\n}\n', '{\n "uuid": "b9d0ff18-fd67-4afa-adbe-ebcb53380804"\n}\n', '{\n "uuid": "70d4d53b-2f06-41be-bb38-47f010cfa40f"\n}\n', '{\n "uuid": "a6d49873-e749-4578-ae9c-e6c6f473535d"\n}\n', '{\n "uuid": "e67efee5-76ad-4812-bb97-016ef9ff87e8"\n}\n', '{\n "uuid": "67886926-b2d9-44fb-b836-26b81c53e5fb"\n}\n', '{\n "uuid": "dcbd4ff8-e3cd-4e03-b12d-5fb3834b0e00"\n}\n', '{\n "uuid": "65c2eaee-5fa2-4b58-a1c3-adeb04d92c71"\n}\n', '{\n "uuid": "2cee4ec9-952e-45c5-91b7-f4f5848c3455"\n}\n', '{\n "uuid": "8e94bf1c-ee5a-483a-a962-d0b9aea48c95"\n}\n', '{\n "uuid": "c1fe17bc-bedf-4c4c-952d-a5921f693d9f"\n}\n', '{\n "uuid": "221456fd-48ca-4826-a8b5-5fa0b23db6e4"\n}\n', '{\n "uuid": "62fda759-b382-44e4-ad7d-d19a952fc1c7"\n}\n', '{\n "uuid": "73faeb91-215e-4e49-8f11-11b98e499cc7"\n}\n', '{\n "uuid": "f3279c45-ebcc-4079-b823-3efe825c7cf8"\n}\n', '{\n "uuid": "b892672b-4510-44f4-b61e-9cccaa52421e"\n}\n', '{\n "uuid": "8926979d-71a7-4171-9389-ddafff89e229"\n}\n', '{\n "uuid": "d97cef59-4862-42ca-b0f2-261f98fd4b6f"\n}\n', '{\n "uuid": "3362ff93-89e4-4889-a2f2-2e03771e86ce"\n}\n', '{\n "uuid": "9f525251-4fe4-4a9c-97b5-2f01d2b37aaf"\n}\n', '{\n "uuid": "036959d4-3179-40f9-bbf3-32274f2cede2"\n}\n', '{\n "uuid": "157f8c22-6214-4e27-ab5d-08d39f96d1d3"\n}\n', '{\n "uuid": "e4bfbf62-7c33-4fd7-a231-47f5ce398041"\n}\n', '{\n "uuid": "a41512c1-3346-4457-a379-64d690ffc2ea"\n}\n', '{\n "uuid": "7bb07cfb-294b-44fa-a8dc-6d283c54409f"\n}\n', '{\n "uuid": "f2297d22-a2d0-47ff-8d65-24c6fe7877a7"\n}\n', '{\n "uuid": "645e255b-4c93-4c8f-9ff2-43da293db660"\n}\n', '{\n "uuid": "9190e370-dfa9-47a6-8cef-8df7ab762433"\n}\n', '{\n "uuid": "83216551-9f1b-48b2-8cd6-fd125a7ce965"\n}\n', '{\n "uuid": "aaddb98c-879b-472d-aa39-1a684ef7d179"\n}\n', '{\n "uuid": "4bd7e2fd-1453-4433-aa9f-bc29d82f5b9d"\n}\n', '{\n "uuid": "b02d65e8-2063-4060-96af-088ec497fc10"\n}\n', '{\n "uuid": "e10e3dd2-83c5-4595-afe4-4145bce79193"\n}\n', '{\n "uuid": "8cb62784-1b5d-4dcc-8342-02ad7d417ca9"\n}\n', '{\n "uuid": "13ef1509-4f69-4426-ac42-cb29a2d0f094"\n}\n', '{\n "uuid": "4d4571d5-69bb-4625-b246-b5eef50aa10d"\n}\n', '{\n "uuid": "75e7a2ca-bfa8-43b9-b33a-f3f927453579"\n}\n', '{\n "uuid": "0a8cc8ff-2039-4873-9e38-afad3e10d726"\n}\n', '{\n "uuid": "189ae75b-4879-4897-9725-f9be17e49844"\n}\n', '{\n "uuid": "ba482468-f45f-4060-a0c1-3ef31bb283c8"\n}\n', '{\n "uuid": "3809f1c7-2f11-487d-bf96-8abf64e08298"\n}\n', '{\n "uuid": "da5ea88b-974d-4238-9654-ac56c657c8b4"\n}\n', '{\n "uuid": "edc3de79-7cf4-42a3-a5f4-b754136a6fd3"\n}\n', '{\n "uuid": "6f5ecd91-537c-4009-8435-6c31ce035d36"\n}\n', '{\n "uuid": "4a33b29d-78ba-468f-8f30-a01b3d9e2a87"\n}\n', '{\n "uuid": "a5a2ef2d-d4a2-48e1-8335-f8c1309328c4"\n}\n', '{\n "uuid": "3d1679da-afdd-4f04-9c16-0aaea4c53d0c"\n}\n', '{\n "uuid": "c4025845-0d4c-4549-acb8-1a249b33e644"\n}\n']
time: 1.0535461902618408
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.