使用python和漂亮的soup4在网页抓取后重复数据

Question

I am trying to scrape data from the Garmin site for golf. I would want to get the name of the golf course and the address but I after running the script. I have noticed that my codes just repeats the first page data over and over again. I also noticed that the page numbers on the website do not start at 1 but at 10 for the second page. How do I go about extracting data from this website and getting all and instead of a repeat of just the first page.

import csv
import codecs
import requests 
from bs4 import BeautifulSoup


courses_list= []
for i in range(10):
    url = "http://sites.garmin.com/clsearch/courses?browse=1&country=US&lang=en&per_page={}".format(i)
    r = requests.get(url)

    soup = BeautifulSoup(r.content)

    g_data2=soup.find_all("div",{"class":"result"})

    for item in g_data2:
     try:
        name= item.contents[3].find_all("div",{"class":"name"})[0].text
        print name
     except:
        name=''
    try:
        address= item.contents[3].find_all("div",{"class":"location"})[0].text
    except:
        address=''


    course=[name,address]
    courses_list.append(course)


with open ('G_Final.csv','a') as file:
    writer=csv.writer(file)
    for row in courses_list:
        writer.writerow([s.encode("utf-8") for s in row])

Answer 1

You discovered the problem.

Then change

url = "http://...?browse=1&country=US&lang=en&per_page={}".format(i)

to

url = "http://...?browse=1&country=US&lang=en&per_page={}".format(i*20)

Answer 2

Just change this, to this:

for i in range(0, 10): url = "http://sites.garmin.com/clsearch/courses?browse=1&country=US&lang=en&per_page={i}"