使用 BeautifulSoup4 解析 Jquery DataTable 中的行

Question

网站： https : //chp.coth.com/entries

我对 BS4 有一些经验，但这让我陷入了困境。 DataTables 通过按下按钮的函数调用执行 XHR 请求，但使用开发人员工具中的“网络”选项卡显示 POST 将转到“ https://chp.coth.com/entries/List ”。

我注意到 from 数据包括页码，我确信有一种方法可以使用 request 发送带有 JSON 字符串的帖子，直到列表末尾，但我不太确定如何去做。 如果没有 Selenium 或类似工具，这可能吗？

Answer 1

此脚本遍历所有页面（1..7，您可以增加此数字以解析更多结果）并向https://chp.coth.com/entries/List发出 POST 请求。 重要的部分是将标题'X-Requested-With'为'XMLHttpRequest'和正确的页码：

import requests
from bs4 import BeautifulSoup

url = 'https://chp.coth.com/entries/List'

data = {
    'num': '',
    'page': '1',
    'sort_on': 'number',
    'sort_type': 'ASC',
}

headers = {'X-Requested-With' : 'XMLHttpRequest'}

for page_num in range(1, 8):
    data['page'] = page_num
    r = requests.post(url, params=data, headers=headers)
    soup = BeautifulSoup(r.text, 'lxml')
    for tr in soup.select('tr'):
        for i, td in enumerate(tr.select('td')):
            if i == 0:
                print('{: >5}'.format(td.text), end='|')
            else:
                print('{: >35}'.format(td.text), end='|')
        print()

印刷：

  501|                          LEADGURIA|                         WESLEY FOX|                     LAURIE GRAYSON|
  502|                         WINCHESTER|                     CHIARA SASSANO|                     LAURIE GRAYSON|
  503|                     MERLIN'S TANGO|                     SYONA MORRISON|                     LAURIE GRAYSON|
  504|               GOLDINO VAN KOEKSHOF|                    COLLEEN GILMORE|                     LAURIE GRAYSON|
  510|                         CRESPO VDL|                     KAREN BANISTER|                     KAREN BANISTER|
  511|                          HERRADURA|                     KAREN BANISTER|                     KAREN BANISTER|
  512|                         SUGAR RUSH|                     HALEY HONEGGER|                    ALEXIA HONEGGER|
  520|                           BONJORNO|      ELISE VANDAMME, DENISE TILLEY|                      DENISE TILLEY|
  521|                            SIR RIO|      CHELSEA MAGNESS, STACEY SWINK|                        LAURA BAKER|
  522|                          CANDY MAN|      CHELSEA MAGNESS, STACEY SWINK|                        LAURA BAKER|

  523|                    SERGEANT PEPPER|      CHELSEA MAGNESS, STACEY SWINK|                        LAURA BAKER|
  524|                          CAYENNE F|         STACEY SWINK, BRUCE BANKER|                        LAURA BAKER|
  525|                            DESMOND|                       STACEY SWINK|                        LAURA BAKER|
  526|                      NIKO'S TAKURA|                   ISABELLA MANCINI|                        LAURA BAKER|
  527|                       THE BIG EASY|                    JESSICA OAKLIEF|                        LAURA BAKER|
  528|                      ZEPHYR’S SONG|                    ELEANOR ROZANCE|                        LAURA BAKER|
  529|                            CALISTO|                       LUCIA WELLSO|                        LAURA BAKER|
  530|                      CHESHIRE JACK|                      REVEN SWANSON|                        LAURA BAKER|
  531|                           KEEPSAKE|                       STACEY SWINK|                        LAURA BAKER|
  532|                           GRACEFUL|                        LAURA BAKER|                        LAURA BAKER|

  535|                   BULL RUN'S AGAPE|                        BRYN SADLER|                         WENDY HAIG|
  536|                       CLIFF AFFAIR|                        BRYN SADLER|                         WENDY HAIG|
  537|                          CINVEN CR|                        BRYN SADLER|                         WENDY HAIG|
  538|                            CEZARRO|                        BRYN SADLER|                         WENDY HAIG|
  539|                  BULL RUN'S LIVING|                        BRYN SADLER|                         WENDY HAIG|
  540|               CHICAGO VAN OVERIS Z|                        BRYN SADLER|                         WENDY HAIG|
  546|                  ALL I EVER WANTED|                    TEGAN ZIMMERMAN|                       RENAE COATES|
  547|                      COASTAL CZONE|TEGAN ZIMMERMAN, RENAE COATES, KAITLIN CAMPBELL|                       RENAE COATES|
  554|                        CALLIGRAPHY|                          AVA UHRIG|                    ALEXIA HONEGGER|
  555|                UTWO DE LA CERISAIE|                     ISABELLA UHRIG|                    ALEXIA HONEGGER|

  556|                  MR CAPTAIN COOK B|       ALEXIA HONEGGER, PAIGE MILES|                    ALEXIA HONEGGER|
  558|             KIRIKOU DES BRIMBELLES|                       JAYDEN SCOTT|                    ALEXIA HONEGGER|
  560|       INDOKTRO K VAN 'T KATTENHEYE|                      SUSAN GRIFFIS|                      SUSAN GRIFFIS|
  561|             DEAUVILLE VAN OVERIS Z|                      SUSAN GRIFFIS|                      SUSAN GRIFFIS|
  563|                            TRISTAN|        ERIC LAMON, GRETCHEN AYCOCK|                    GRETCHEN AYCOCK|
  564|                            TRILOGY|                    GRETCHEN AYCOCK|                    GRETCHEN AYCOCK|
  566|                  EMPIRE BEAR BONES|                     RACHEL MALONEY|                     RACHEL MALONEY|
  567|                     WINGS SUBLIEME|                     RACHEL MALONEY|                     RACHEL MALONEY|
  571|                     J.E.S. CIBELLE|                    ASHLEY BUCHANAN|                    JOHN MC CONNELL|
  572|                   CHIANTI CLASSICO|                    ASHLEY BUCHANAN|                    JOHN MC CONNELL|

  573|                        POLAR LIGHT|                    ASHLEY BUCHANAN|                    JOHN MC CONNELL|
  574|                           BYKARLOS|                       KATHY KRAMER|                    ALEXIA HONEGGER|
  576|                             VONNOH|                   SABRINA OHLROGGE|                    ALEXIA HONEGGER|
  577|                        JUST SAY SO|                      HANNAH WRIGHT|                     LAURIE GRAYSON|
  580|                         BURGUNDY B|                     AMELIA JOHNSON|                    ALEXIA HONEGGER|
  583|                      PITCH PERFECT|                    EMILY SCHOELZEL|                    REBECCA JOHNSON|
  584|                           CATANA S|                    SHELBY BURLESON|                          AMY HENZE|
  585|                           CALLE 87|     ASHLEY RHEINGOLD, SARAH WATSON|                          AMY HENZE|
  586|                             AMEDEO|                      JULIA FISCHER|                          AMY HENZE|
  589|                               ITON|                  ISABELLA LOMBARDO|                    REBECCA JOHNSON|

  590|             PLEASANT VALLEY SUNDAY|                        SARAH YOUNG|                     JANE SCHWEIGER|
  596|                 GODS DREAM CATCHER|         KAIYA KOLB, KAREN BANISTER|                     KAREN BANISTER|
  598|                           MAGNOLIA|                     KAREN BANISTER|                     KAREN BANISTER|
  606|                          HIGH TIME|                       ANNIE GRABER|                       ANNIE GRABER|
  607|                     RINGO'S A STAR|                     RILEY GILBRETH|                    ALEXIA HONEGGER|
  608|                            COURAGE|       MEGAN SOUTHAM, EMERY VOLKERT|                    CINDY CRUCIOTTI|
  609|                              NEIGH|                     GRACE HAMILTON|                    CINDY CRUCIOTTI|
  610|                    INDEPENDENCE KF|    ELIZABETH LATSIS, MEGAN SOUTHAM|                    CINDY CRUCIOTTI|
  611|                       THREE RIVERS|   JENNIFER WALLEN, KELLI CRUCIOTTI|                    CINDY CRUCIOTTI|
  612|                              MARGO|  MEGAN SOUTHAM, CASSANDRA RAGSDALE|                    CINDY CRUCIOTTI|

  613|                          CEDRIC 42|                    HANNA GILHOOLEY|                    CINDY CRUCIOTTI|
  614|                      MISS SCARLETT|GLORIA FOUNTAIN JAMES, MARIELLA  MILLER|                    CINDY CRUCIOTTI|
  615|                         ISHANA III|          MEGAN SOUTHAM, JENNA KOLB|                    CINDY CRUCIOTTI|
  616|                          REX ROMEO|    ELIZABETH LATSIS, MEGAN SOUTHAM|                    CINDY CRUCIOTTI|
  617|                                Q-T|    KELLI CLEVENGER, CAMBELL BOWERS|                    CINDY CRUCIOTTI|
  618|                        CAYMAN A.L.|ABIGAIL GRACE KELLEY, MEGAN SOUTHAM|                    CINDY CRUCIOTTI|
  619|                      ALL INCLUSIVE|               ABIGAIL GRACE KELLEY|                    CINDY CRUCIOTTI|
  620|                          MARGARITA|                 EMILY ELEK-BURTARD|                    CINDY CRUCIOTTI|
  621|                   SOUTHERN MARTINI|                      JOANNE JANSON|                    ALEXIA HONEGGER|
  623|                              BALDO|                        KATIE GLASS|                      KRISTIN JACOB|