調用 API 時如何限制 python pandas 的速率/速度？

Question

I'm using the pandas apply function with a function to call an API and populate a column in a dataframe. API 的 QPS 為 500。如果超過限制，我將受到限制並最終被阻止。 如何限制應用 function 填充 dataframe 的速率/速度？

Answer 1

您必須跟蹤您在最后一秒（或您的速率限制指定的任何時間范圍內）發出了多少請求。 一旦達到極限，或者就在此之前，你將不得不等待一段時間。

import time
import pandas
import datetime
import requests

# Generate some test data
df = pandas.DataFrame([{"todo_id": i} for i in range(100)])

# Define variables to keep track of throttling
count = 0
last_reset_time = datetime.datetime.now()
rate_limit = 5  # Max number of requests per second


# Define a function to enrich the data
def enrich(row: pandas.Series) -> pandas.Series:

    # Access the global variables
    global count
    global last_reset_time

    # Wait until we are below the rate limit
    if count >= rate_limit:
        while True:
            now = datetime.datetime.now()
            if (now - last_reset_time) <= datetime.timedelta(seconds=1):
                print("Waiting...")
                time.sleep(0.1)
            else:
                count = 0
                last_reset_time = datetime.datetime.now()
                break

    # Enrich the data
    todo_id = row['todo_id']
    response = requests.get(f"https://jsonplaceholder.typicode.com/todos/{todo_id}")
    data = response.json()
    for key, value in data.items():
        row[key] = value

    # Increment the counter
    count += 1

    # Return the row, which will be put in the dataframe
    return row


# Apply the function
df.apply(enrich, axis=1)