I have a HTML Page with multiple divs like:
<td class="b-list__main">
<a data-gtm="Page B list" href="C.php?bsn=31888&snA=773&tnum=2" class="b-list__main__title">【info】10/23 develop note-new character</a><span class="b-list__main__icon"><i title="有圖片" class="material-icons icon-photo"></i></span>
</td>
I am new to python and BeautifulSoup, I am trying to get all the urls from this class. I have tried:
for lastpage in root.find_all("td", class_="b-list__main"):
print(lastpage.p)
output:
<p class="b-list__main__title" data-gtm="Page B list" href="C.php?bsn=31888&snA=773&tnum=2">【info】10/23 develop note-new character</p>
<p class="b-list__main__title" data-gtm="Page B list" href="C.php?bsn=31888&snA=774&tnum=1">【Q】alient team choice</p>
<p class="b-list__main__title" data-gtm="Page B list" href="C.php?bsn=31888&snA=772&tnum=1">【Q】lock account question</p>
My ideal output is to get the biggest number, for example 774. But I am doing one step at the time, just try to get url first then number.
C.php?bsn=31888&snA=773&tnum=2
C.php?bsn=31888&snA=774&tnum=1
C.php?bsn=31888&snA=772&tnum=1
I also tried:
for lastpage in root.find_all("td", class_="b-list__main"):
link = lastpage.fine('p',href=True)
if link is None:
continue
print(lastpage.p['href'])
but getting TypeError: 'NoneType' object is not subscriptable
Any help is appreciated, thanks.
My code:
import bs4
import re
def getData(url):
request = req.Request(url, headers={
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 "
})
with req.urlopen(request) as response:
data = response.read().decode("utf-8")
root = bs4.BeautifulSoup(data, "html.parser")
for lastpage in root.find_all("td", class_="b-list__main"):
print(lastpage.p)
url = "https://forum.gamer.com.tw/B.php?bsn=31888"
getData(url)
I have never seen a p
tag with a href
attribute, but if that is how the html code looks like, you can try something like this:
from bs4 import BeautifulSoup
import re
html = """
<p class="b-list__main__title" data-gtm="Page B list" href="C.php?bsn=31888&snA=773&tnum=2">【info】10/23 develop note-new character</p>
<p class="b-list__main__title" data-gtm="Page B list" href="C.php?bsn=31888&snA=774&tnum=1">【Q】alient team choice</p>
<p class="b-list__main__title" data-gtm="Page B list" href="C.php?bsn=31888&snA=772&tnum=1">【Q】lock account question</p>
"""
root = BeautifulSoup(html,'html5lib')
links_lst = []
for lastpage in root.find_all("p"):
links_lst.append(lastpage['href'])
Output:
>>> links_lst
['C.php?bsn=31888&snA=773&tnum=2', 'C.php?bsn=31888&snA=774&tnum=1', 'C.php?bsn=31888&snA=772&tnum=1']
In order to find the largest number, you can use a bit of regex
. Just add these lines to the code provided above:
pattern = re.compile('(?<=snA=).*\d{3}')
num_lst = []
for link in links_lst:
num_lst.append(int(pattern.findall(link)[0]))
print(f"Largest Number = {max(num_lst)} , Full link = {links_lst[num_lst.index(max(num_lst))]}")
Output:
Largest Number = 774 , Full link = C.php?bsn=31888&snA=774&tnum=1
Edit:
Here is the full code:
import bs4
import re
from urllib import request as req
links_lst = []
def getData(url):
request = req.Request(url, headers={
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 "
})
with req.urlopen(request) as response:
data = response.read().decode("utf-8")
root = bs4.BeautifulSoup(data, "html.parser")
for lastpage in root.find_all("div", class_="b-list__tile"):
try:
links_lst.append(lastpage.p['href'])
except:
pass
pattern = re.compile('(?<=snA=).*\d{3}')
num_lst = []
for link in links_lst:
num_lst.append(int(pattern.findall(link)[0]))
print(f"Largest Number = {max(num_lst)} , Full link = {links_lst[num_lst.index(max(num_lst))]}")
url = "https://forum.gamer.com.tw/B.php?bsn=31888"
getData(url)
Output:
Largest Number = 774 , Full link = C.php?bsn=31888&snA=774&tnum=1
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.