使用循环抓取网站时丢失数据

Question

Beginner coder here trying to do a simple website scrape. I want to pull item attributes from multiple pages of a search result. I can do that, but my issue is that items towards the end of each page seem to be missing.

Is it a simple error with my loop/counters? The example code below should just be printing x as the xth search result.

import requests
from bs4 import BeautifulSoup
import xlwt

headers = {'user-agent': 'Mozilla/5.0'}
pagelimit = 60 #number of results on page
startoffset = 0 #starting offset  (no. of items


def extract(soup,count):
    x = count
    for div in soup.findAll("div", "result-item standard"):
        print(x)
        x = x+1

offset = startoffset
count = 1
for i in range(0,10):
    url = "http://www.carsales.com.au/cars/results?offset=" + \
    str(offset) + \
    "&q=%28Service%3D%5BCarsales%5D%26%28%28SiloType%3D%5BDealer%20" + \
    "used%20cars%5D%7CSiloType%3D%5BDemo%20and%20near%20new%" + \
    "20cars%5D%29%7CSiloType%3D%5BPrivate%20seller%20cars%5D%29%29" + \
    "&sortby=~Price&limit=" + \
    str(pagelimit) + "&cpw=1"

    r = requests.get(url, headers)
    soup = BeautifulSoup(r.text, "html.parser")
    extract(soup,count)

    offset = str(i*pagelimit+int(startoffset))
    count = count + pagelimit

Answer 1

Your code makes two assumptions which might lead to missing results

The first assumption is that every page returns the maximum number of results ( pagelimit ) - which the final page is unlikely to. You should have the extract method return the final value of x:

def extract(soup,count):
    x = count
    for div in soup.findAll("div", "result-item standard"):
        print(x)
        x = x+1
    return x

Then you should replace count = count + pagelimit with something like count = extract(soup,count)

You can also then use this number to set the offset.

The second assumption is that there will always be at least 10 pages of cars. If there are less than 10 pages, your code may behave strangely when you loop beyond the end of the results list.