在python / BeautifulSoup中进行网络爬网时将值附加到可迭代的对象（还想了解多线程）

Question

I am coming here with a bug on my very first real python program (ie not something out of codeacademy). I am an avid user of R and I built out a bunch of web crawling/scraping tools using the XML packages. Unfortunately, I reached the point where R is just not ideal for some of the things that I am trying to do, so I am building some of these tools in Python and Hopefully you can help. If there are any glaring python-specific coding best-practices that I'm neglecting, I would appreciate a heads-up as well.

I need to be able to append values to my iterable. At each step of the iteration in the code below, the entire object links_child gets added to my the links object. I actually need each link within links_child to be added separately to the object, rather than all as one entry. This will keep growing and growing links and the iteration will only break when I reach a specified number of websites (100 in the code below). Do any of you know how I can extract the bs4 items line-by-line and add them to my iterable object? The error I get is as follows:

AttributeError: 'ResultSet' object has no attribute 'get'

Also, I ultimately want this crawler to be a heck of a lot faster. What types of mutli-threading options (if any) do I have with beautifulsoup? Should I switch libraries? Are there any low-hanging fruits I can pick here that would boost my speed? It would be perfect if there was a simple way to have 5-10 threads doing this crawl at once and updating the same dictionary object, but that is probably just a fantasy.

from urllib import urlopen
from bs4 import BeautifulSoup
import re

base_site = "http://www.tasq.com"
page = urlopen(base_site).read()
soup = BeautifulSoup(page)
links = soup.find_all('a')
start_slash = re.compile('/')
link_db = {}
# iterate through all links on the homepage
for link in links:
    # print for debugging purposes
    print link
    # pull out the hrefs from the current link
    fullLink = str(link.get('href'))
    # print for debugging purposes
    print fullLink
    # see if the link is valid using regex
    check_start = start_slash.match(fullLink)
    # if the link is not valid, concatenate the domain
    if check_start <> None:
        fullLink = base_site + fullLink
    # fi the link is already stored as a key in the dict, skip it
    if fullLink in link_db:
        next
    # connect to the full link (O operation)
    page_child = urlopen(fullLink).read()
    # create bs4 object out of the opened page
    soup_child = BeautifulSoup(page_child)
    # insert the link as the key and some text string as the value
    link_db[fullLink] = 'example'
    # find all links on current page and save them in object
    links_child = soup_child.find_all('a')
    # (THIS IS THE SOURCE OF THE ERROR)append object with links to the iterable
    links.append(links_child)
    # break code if link_db gets to specified length
    if len(link_db) == 100:
        break

Answer 1

You're cycling over links and you're appending over it as well. Eventually it hits the first links_child you added, which is a list of links not a Tag object and therefore doesn't have a get attribute.

Append the links_child to another variable and it works fine. You can also use extend instead of append to add the contents of links_child to links but it hits another problem further on trying to read a relative URL ../contact/contact-form.php which you're not accounting for.

There's several ways to do multiprocessing in Python, the most popular is multiprocessing as it gives you a nice API to work with along with spawning processes rather than threads which would fully make use of the multiple cores in the CPU.

There's a number of ways you can approach multiprocessing in this example. You could, for instance, define your main loop as a function add create a Pool of workers to work through it. Something like this:

from urllib import urlopen
from bs4 import BeautifulSoup
import re
import multiprocessing

def work(link):
    link_db = {}
    start_slash = re.compile('/')
    print link
    fullLink = link.attrs.get('href', None)
    check_start = start_slash.match(fullLink)
    if check_start != None:
        fullLink = base_site + fullLink
    page_child = urlopen(fullLink).read()
    soup_child = BeautifulSoup(page_child)
    link_db[fullLink] = 'example'
    return link_db

if __name__ == '__main__':
    base_site = "http://www.tasq.com"
    page = urlopen(base_site).read()
    soup = BeautifulSoup(page)
    links = soup.find_all('a')
    link_dbs = []
    pool = multiprocessing.Pool(processes=4)
    result = pool.map_async(work, links)
    link_dbs.extend( result.get() )
    print link_dbs

Take this as guideline though, I simplified your function to make it clearer. Hopefully this will get you on track.