简体   繁体   中英

python scrape page + next page loop from url or element button

It just prints the first page does not go and prints the other page where is the problem? I'm new to Python

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
import requests

for num in range(1,6):     #Number of pages plus one
    url = (f'https://test.com/%D9%84-%8%B6/{num}')
    html = urlopen(url)
    r = requests.get(url)
    bs = BeautifulSoup(html,'html.parser')

You want to be doing something like so:

from bs4 import BeautifulSoup
import requests

for num in range(1, 6):  # Number of pages plus one
    url = f"https://google.com/{num}"
    r = requests.get(url)
    bs = BeautifulSoup(r.text, "html.parser")
    print(bs)

urllib.request import urlopen is not needed for this.


Regarding the comments below, here is getting the title of my website with BeautifulSoup :

from bs4 import BeautifulSoup
import requests

url = "https://self.st"  # You would have your number here.
r = requests.get(url)
bs = BeautifulSoup(r.text, "html.parser")

title = bs.find("div", {"class": "title"})

print(title)
print(title.text)

Outputs:

<div class="title">felipe faria</div>
felipe faria

在此处输入图像描述

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM