使用 BeautifulSoup 進行抓取移動到下一頁

Question

我需要從網站上抓取內容（只是標題）。 我只做了一頁，但我需要對網站上的所有頁面都做。 目前，我的做法如下：

import bs4, requests
import pandas as pd
import re

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}
    
    
r = requests.get(website, headers=headers)
soup = bs4.BeautifulSoup(r.text, 'html')


title=soup.find_all('h2')

我知道，當我移動到下一頁時，url 更改如下：

website/page/2/
website/page/3/
... 
website/page/49/
...

我嘗試使用 next_page_url = base_url + next_page_partial 構建遞歸函數，但它沒有移動到下一頁。

if soup.find("span", text=re.compile("Next")):
    page = "https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina/page/".format(page_num)
    page_num +=10 # I should scrape all the pages so maybe this number should be changed as I do not know at the beginning how many pages there are for that section
    print(page_num)
else:
    break

我關注了這個問題（和答案）： 移動到下一頁使用 BeautifulSoup 進行抓取

如果您需要更多信息，請告訴我。 非常感謝

更新代碼：

import bs4, requests
import pandas as pd
import re

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}

page_num=1
website="https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina"

while True:
  r = requests.get(website, headers=headers)
  soup = bs4.BeautifulSoup(r.text, 'html')


  title=soup.find_all('h2')

  if soup.find("span", text=re.compile("Next")):
      page = f"https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina/page/{page_num}".format(page_num)
      page_num +=10
  else:
      break

Answer 1

如果您使用f"url/{page_num}"則刪除format(page_num) 。

您可以在下面使用任何您想要的東西：

page = f"https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina/page/{page_num}"

或者

page = "https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina/page/{}".format(page_num)

祝你好運！

---

最終答案將是這樣的：

import bs4, requests
import pandas as pd
import re

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}

page_num=1
website="https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina"

while True:
  r = requests.get(website, headers=headers)
  soup = bs4.BeautifulSoup(r.text, 'html')


  title=soup.find_all('h2')

  if soup.find("span", text=re.compile("Next")):
      website = f"https://catania.liveuniversity.it/notizie-catania-cronaca/cronacacatenesesicilina/page/{page_num}"
      page_num +=1
  else:
      break