Web爬虫页面迭代

Question

I have written this code that goes to webMD and so far extracts all the link from each sub category in the message boards. What I was to do next is to make the program go through all the pages of the subcategory link. I have tried many thing but I always face a problem any idea?

import bs4 as bs
import urllib.request
import pandas as pd


source = urllib.request.urlopen('https://messageboards.webmd.com/').read()

soup = bs.BeautifulSoup(source,'lxml')


df = pd.DataFrame(columns = ['link'],data=[url.a.get('href') for url in soup.find_all('div',class_="link")])
lists=[]
for i in range(0,33):
    link = (df.link.iloc[i])
    source1 = urllib.request.urlopen(link).read()
    soup1 = bs.BeautifulSoup(source1,'lxml')

Answer 1

I've used Python and Wget to do a similar task in the past. See Wget documentation here . You can look into the source to get an idea of how it works.

Basically you can do the following. See the follwoing Pseudo code

alreadyDownloadedUrls = []
currentPageUrls = []

def pageDownloader('url'):
    downaload the given URL
    append the url to 'alreadyDownloadedUrls' list
    return the given URL

def urlFinder('inputPage'): 
    finds and returns all the URL of the input page in a list

def urlFilter ('inputUrl or list of URLs'):
    check if the input list of URLs are already in the 'alreadyDownloadedUrls' list, 
    if not appends that URL to a local list variable and returns

def controlFunction(firstPage):
    Download the first page
    firstPageDownload = pageDownloader(firstPage)
    foundUrls = urlFinder (firstPageDownload)
    validUrls = urlFilter(foundUrls)
    currentlyWorkingList = []
    for ( the length of validUrls):
         downloadUrl = pageDownloader(aUrl from the list)
         append to currentlyWorkingList
    for (the lenght of currentlyWorkingList):
        call controlFunction() recursively

However, recursively calling will result you to download the whole internet. So you have to validate URLs and see if its from the parent domain or subdomain. You can do that in the urlFilterFunction .

Also you have to add some more validation to check if you are downloading the same link with a hash tag at the end of the url. Unless your programme will think this and this URLs are pointing to a different pages.

You may also introduce a depth limit as in Wget

Hope this clears the idea to you.