网页抓取发现没有移动到下一个项目

Question

from bs4 import BeautifulSoup
import requests


def kijiji():
    source = requests.get('https://www.kijiji.ca/b-mens-shoes/markham-york-region/c15117001l1700274').text
    soup = BeautifulSoup(source,'lxml')
    b = soup.find('div', class_='price')
    for link in soup.find_all('a',class_ = 'title'):
        a = link.get('href')
        fulllink = 'http://kijiji.ca'+a
        print(fulllink)
        b = soup.find('div', class_='price')
        print(b.prettify())
kijiji()

Usage of this is to sum up all the different kinds of items sold in kijiji and pair them up with a price. But I can't seem to find anyway to increment what beautiful soup is finding with a class of price, and I'm stuck with the first price. Find_all doesn't work either as it just prints out the whole blob instead of grouping it together with each item.

Answer 1

If you have Beautiful soup 4.7.1 or above you can use following css selector select() which is much faster.

code:

import requests
from bs4 import BeautifulSoup

res=requests.get("https://www.kijiji.ca/b-mens-shoes/markham-york-region/c15117001l1700274").text
soup=BeautifulSoup(res,'html.parser')
for item in soup.select('.info-container'):
    fulllink = 'http://kijiji.ca' + item.find_next('a', class_='title')['href']
    print(fulllink)
    price=item.select_one('.price').text.strip()
    print(price)

---

Or to use find_all() use below code block

import requests
from bs4 import BeautifulSoup

res=requests.get("https://www.kijiji.ca/b-mens-shoes/markham-york-region/c15117001l1700274").text
soup=BeautifulSoup(res,'html.parser')
for item in soup.find_all('div',class_='info-container'):
    fulllink = 'http://kijiji.ca' + item.find_next('a', class_='title')['href']
    print(fulllink)
    price=item.find_next(class_='price').text.strip()
    print(price)

Answer 2

Congratulations on finding the answer. I'll give you another solution for reference only.

import requests
from simplified_scrapy.simplified_doc import SimplifiedDoc
def kijiji():
  url = 'https://www.kijiji.ca/b-mens-shoes/markham-york-region/c15117001l1700274'
  source = requests.get(url).text
  doc = SimplifiedDoc(source)
  infos = doc.getElements('div',attr='class',value='info-container')
  for info in infos:
    price = info.select('div.price>text()')
    a = info.select('a.title')
    link = doc.absoluteUrl(url,a.href)
    title = a.text
    print (price)
    print (link)
    print (title)
kijiji()

Result:

$310.00
https://www.kijiji.ca/v-mens-shoes/markham-york-region/jordan-4-oreo-2015/1485391828
Jordan 4 Oreo (2015)
$560.00
https://www.kijiji.ca/v-mens-shoes/markham-york-region/yeezy-boost-350-yecheil-reflectives/1486296645
Yeezy Boost 350 Yecheil Reflectives
...

Here are more examples: https://github.com/yiyedata/simplified-scrapy-demo/tree/master/doc_examples

Answer 3

from bs4 import BeautifulSoup
import requests


def kijiji():
    source = requests.get('https://www.kijiji.ca/b-mens-shoes/markham-york-region/c15117001l1700274').text
    soup = BeautifulSoup(source,'lxml')
    b = soup.find('div', class_='price')
    for link in soup.find_all('a',class_ = 'title'):
        a = link.get('href')
        fulllink = 'http://kijiji.ca'+a
        print(fulllink)
        print(b.prettify())
        b = b.find_next('div', class_='price')
kijiji()

Was stuck on this for an hour, as soon as I posted this on stack I immediately came up with an idea, messy code but works!