在网站上抓取多个页面

Question

I'm trying to scrape a list of all the coaching instiututes on thsi URL: https://www.sulekha.com/entrance-exam-coaching/delhi

The following is my Python code:

import bs4
from urllib.request
import urlopen as uReq
from bs4
import BeautifulSoup as soup

my_url = 'https://www.sulekha.com/entrance-exam-coaching/delhi'

uClient = uReq(my_url)
page_html = uClient.read()
uClient.close() x

page_soup = soup(page_html, "lxml")


insti = page_soup.findAll("div", {"class": "list-title"})

filename = "entrance_institutes.csv"

f = open(filename, "w")
headers = "Institute \n"
f.write(headers)

for ins in insti:
    ins_name = ins.div.a["title"]

f.write(ins_name + "\n")

f.close()

This code runs fine. Attached is the image of the csv it generates. How should I go about scraping all the listings one page after the other ?

Thanks

Output csv

Answer 1

I'm not 100% sure what you mean. If you're asking how to fix the bug in your code then you need to change your loop to:

for ins in insti:
    ins_name = ins.div.a["title"]
    f.write(ins_name + "\n")

As your code is you loop through the everything and write the last one due to the write not being in the loop.

However if you're asking how to take that list and then scrap those then that's more involved and for starters you need to save the URL rather than the title but I'm going to leave the rest to you because that kind of sounds like homework.