InvalidSchema：未找到“{'url':

Question

賞金將在 6 天后到期。 此問題的答案有資格獲得+100聲望賞金。 Arun.K正在從信譽良好的來源尋找答案：

能夠通過代碼在分頁中傳遞下一個鏈接，並從所有頁面中獲取數據

我正在嘗試使用 pyspark 進行分頁並出現以下錯誤。 我的分頁鏈接在 header 中作為鍵 [Link] 和值 [rel="next"]。 錯誤顯示在此行 r1 = requests.get(response.links['next'])。問題是傳遞的“下一個”URL 中缺少 baseURL。

getURL = 'https://api.xxx.com/v3/direct-access/abc'
baseURL = 'https://api.xxx.com/v3/direct-access'
headers = {
"accept" : "application/json",
"Content-Type": "application/json",
"Authorization": "Bearer " + str(token)
 }
results = []

response = requests.get(getURL, headers=headers)
r = response.json()  

for i in r:  
results.append(i) 

while response.links['next']: ## != response.links['last']:  
r1 = requests.get(response.links['next'])  
r = r1.json()  
for i in r:  
    results.append(i)
 
Error: InvalidSchema: No connection adapters were found for "{'url': '/abc?  action=next&next_page=%28id%2Ccompletionid%29+%3C+%28840430000754002%2C840430413029241%29&pagesize=10000', 'rel': 'next'}"

如何將 baseURL 和 url 合並為一個鏈接並將其傳遞到 while 循環中？ 像下面這樣的東西

https://api.xxx.com/v3/direct-access/abc?action=next&next_page=%28id%2Ccompletionid%29+%3C+%28840430000754002%2C840430413029241%29&pagesize=10000

Answer 1

像這樣：

while response.links['next']:
    next_link = response.links['next']['url']
    r1 = requests.get(baseURL + next_link, headers=headers)
    r = r1.json()
    for i in r:
        results.append(i)

Answer 2

您的代碼有幾個問題。

1. response.links['next']是一個字典{'url': ...} 。 requests.get(...)需要一個 URL 字符串。

# requests.get(response.links['next'])
requests.get(response.links['next']['url'])
# requests.get(baseURL + response.links['next']['url'])  # With baseURL

2. response沒有被修改，導致死循環。

while response...
    # r1 = requests.get(response.links['next'])
    response = requests.get(baseURL + response.links['next']['url'])

3. 對於最后一個鏈接， 'next'將不存在。

# while response.links['next']:
while response.links.get('next'):

---

最小的，可重現的例子：

import requests

getURL = 'https://api.github.com/users/acjh/repos'
baseURL = ''
headers = {}
results = []

response = requests.get(getURL, headers=headers)
r = response.json()

for i in r:
    results.append(i)

while response.links.get('next'):
    response = requests.get(baseURL + response.links['next']['url'])
    r = response.json()
    for i in r:
        results.append(i)

assert len(results) == requests.get(getURL[:-6]).json()['public_repos']