使用 Python 的 requests.post 函数抓取动态 HTML 时，如何指定“页码”？

Question

我正在尝试学习如何使用 Python 中的requests库来抓取带有动态 HTML 的网页。 我已经能够获取 HTML，但我无法弄清楚如何实际获取不同页面的数据。 如果您注意到图像底部有页码：

1. 如何使用requests.post指定页码
2. 我怎么知道有多少页可用？ 它会像页面范围一样简单，然后try和except吗？
3. 我怎么知道我的data字典可以在requests.post使用哪些参数？

这是我访问请求的方式：

import requests
r = requests.post('https://search.aap.org/nicu/', 
data = {'SearchCriteria.Level':'1', 'X-Requested-With':'XMLHttpRequest'}) #key:value
print(r.text)

这是输出：

https://pastebin.com/raw/HMBZWR36

这是它说页码的部分：

# <div id="myPager">
#     <div class="pagination-container"><ul class="pagination blue-pager"><li class="blue-pager active"><span>1</span></li></ul></div>

# </div>

Answer 1

该页面上的数据是使用 javascript 动态加载的，因此您不能仅通过来自该页面的请求来获取它。 在浏览器中使用 Developer 标签，您可以拦截生成数据的 xhr 并将其用于请求。 这就是这种情况下的样子。

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0',
    'Accept': '*/*',
    'Accept-Language': 'en-US,en;q=0.5',
    'X-Requested-With': 'XMLHttpRequest',
    'Connection': 'keep-alive',
    'Referer': 'https://search.aap.org/nicu/',
}

params = (
    ('Model', 'X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]'),
    ('jsonSearchVmString', '{"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":2,"LevelAssigner":null,"BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}'),
    ('page', '2'),
    ('class', 'blue-pager'),
    ('X-Requested-With', 'XMLHttpRequest'),
    ('_', '1603318486561'),
)

response = requests.get('https://search.aap.org/nicu/', headers=headers, params=params)

参数之一是('page', '2') ; 如果您将请求包装在适当范围内的for循环中，您应该能够在所有页面上执行requests.get() 。

Answer 2

在这种情况下，您不必使用 POST 请求。 如果在单击下一页时查看 URL，您将看到所有可以更改的参数。 此外，要获取所有页面，您可以解析 HTML 并查找<a rel="next">标签。 如果该标记不存在，则您位于最后一页（尝试将"LevelAssigner"参数从3更改为其他值）：

import requests
from bs4 import BeautifulSoup


url = 'https://search.aap.org/nicu/?Model=X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]&jsonSearchVmString={"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":0,"LevelAssigner":"3","BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}&class=blue-pager&X-Requested-With=XMLHttpRequest&page='

page = 1
while True:
    print('Page {}..'.format(page))

    soup = BeautifulSoup(requests.get(url + str(page)).content, 'html.parser')

    for row in soup.select('.row.font-700'):
        print(row.get_text(strip=True, separator=' '))

    next_page = soup.select_one('a[rel="next"]')
    if not next_page:
        break

    page += 1

印刷：

Page 1..
ANDERSON REGIONAL MEDICAL CENTER Meridian, MS 39301
Ann & Robert H. Lurie Children's Hospital of Chicago Chicago, IL 60611
BAYSTATE MEDICAL CENTER Springfield, MA 01199
Beacon Children's Hospital South Bend, IN 46601-1033
BEN TAUB GENERAL HOSPITAL Houston, TX 77030
Page 2..
BETH ISRAEL DEACONESS MEDICAL CENTER Boston, MA 02215
BRIGHAM AND WOMEN'S HOSPITAL Boston, MA 02115
CHILDREN'S HOSPITAL OF NEW ORLEANS New Orleans, LA 70118
CLEVELAND CLINIC formally MERIDIA HILLCREST HOSPITAL Mayfield Heights, OH 44124
EAST JEFFERSON GENERAL HOSPITAL Metairie, LA 70006
Page 3..
GEORGETOWN UNIVERSITY HOSPITAL Washington, DC 20007
GREENWICH HOSPITAL Greenwich, CT 06830
HELEN DEVOS CHILDREN'S HOSPITAL Grand Rapids, MI 49503
INOVA LOUDOUN HOSPITAL Leesburg, VA 20176
JACOBI MEDICAL CENTER Bronx, NY 10461
Page 4..
LAKEVIEW REGIONAL MEDICAL CENTER HOSPITAL Covington, LA 70433
LOYOLA UNIVERSITY MEDICAL CENTER Maywood, IL 60153
MELROSE WAKEFIELD HOSPITAL Melrose, MA 02176
MERCYHEALTH HOSPITAL ROCKTON AVENUE CAMPUS formally ROCKFORD HEALTH SYSTEM Rockford, IL 61103
RUSH FOUNDATION HOSPITAL Meridian, MS 39301
Page 5..
Shetal Shah Valhalla, NY 10595
SLIDELL MEMORIAL HOSPITAL Slidell, LA 70458
SOUTHERN NEW HAMPSHIRE MEDICAL CENTER Nashua, NH 03061
St Mary Medical Center Apple Valley, CA 92307
ST. AGNES HOSPITAL Baltimore, MD 21229
Page 6..
THE CHILDREN'S HOSPITAL AT NAVICENT HEALTH Macon, GA 31208-6000
TOURO INFIRMARY New Orleans, LA 70115
TULANE LAKESIDE HOSPITAL Metairie, LA 70001
UNIVERSITY MEDICAL CENTER -  SUNY STONY BROOK Stony Brook, NY 11794-8111
UNIVERSITY OF IOWA CHILDREN'S HOSPITAL Iowa City, IA 52242
Page 7..
UNIVERSITY OF LOUISVILLE SCHOOL OF MEDICINE Louisville, KY 40232-5070
UNIVERSITY OF MISSISSIPPI MEDICAL CENTER Jackson, MS 39216-4505
UNIVERSITY OF ROCHESTER MEDICAL CENTER Rochester, NY 14642
UNIVERSITY OF WASHINGTON MEDICAL CENTER Seattle, WA 98195
WakeMed Cary Hospital Cary, NC 27518
Page 8..
WakeMed North Family Health & Women’s Hospital Raleigh, NC 27614
WakeMed Raleigh Hospital Raleigh, NC 27610
Wellstar Kennestone Hospital NICU Marietta, GA 30066
WILLIS KNIGHTON CENTER FOR WOMEN'S HEALTH Shreveport, LA 71118
WOMAN'S HOSPITAL Baton Rouge, LA 70817