How to write a python script to search a website html for matching links

Question

I am not too familiar with python and have to write a script to perform a host of functions. Basically the module i still need is how to check a website code for matching links provided beforehand.

Answer 1

Matching links what? Their HREF attribute? The link display text? Perhaps something like:

from BeautifulSoup import BeautifulSoup, SoupStrainer
import re
import urllib2

doc = urllib2.urlopen("http://somesite.com").read()
links = SoupStrainer('a', href=re.compile(r'^test'))
soup = [str(elm) for elm in BeautifulSoup(doc, parseOnlyThese=links)]
for elm in soup:
    print elm

That will grab the HTML content of somesite.com and then parse it using BeautifulSoup, looking only for links whose HREF attribute starts with "test". It then builds a list of these links and prints them out.

You can modify this to do anything using the documentation .

Answer 2

Generally, you use urllib , urllib2 (htmllib etc) for programming web in Python. you could also use mechanize , curl etc. Then for processing HTML and getting links, you would want to use parsers like BeautifulSoup .

Answer 3

try scrapy , the most comprehensive web extraction framework.

http://scrapy.org