Python3刮板。 直到最后才解析xpath

Question

I'm using lxml.html module

from lxml import html   

page = html.parse('http://directory.ccnecommunity.org/reports/rptAccreditedPrograms_New.asp?sort=institution')

# print(page.content)

unis = page.xpath('//tr/td[@valign="top" and @style="width: 50%;padding-right:15px"]/h3/text()')

print(unis.__len__())

with open('workfile.txt', 'w') as f:
    for uni in unis:
        f.write(uni + '\n')

The website right here ( http://directory.ccnecommunity.org/reports/rptAccreditedPrograms_New.asp?sort=institution#Z ) is full of universities.

The problem is that it parses till the letter 'H' (244 unis). I can't understand why, as I see it parses all the HTML till the end.

I also documented my self that 244 is not a limit of a list or anything in python3.

Answer 1

That HTML page simply isn't HTML, it's totally broken. But the following will do what you want. It uses the BeautifulSoup parser.

from lxml.html.soupparser import parse
import urllib

url = 'http://directory.ccnecommunity.org/reports/rptAccreditedPrograms_New.asp?sort=institution'
page = parse(urllib.request.urlopen(url))
unis = page.xpath('//tr/td[@valign="top" and @style="width: 50%;padding-right:15px"]/h3/text()')

See http://lxml.de/lxmlhtml.html#really-broken-pages for more info.

Answer 2

For web-scraping i recommend you to use BeautifulSoup 4 With bs4 this is easily done:

from bs4 import BeautifulSoup
import urllib.request

universities = []
result = urllib.request.urlopen('http://directory.ccnecommunity.org/reports/rptAccreditedPrograms_New.asp?sort=institution#Z')

soup = BeautifulSoup(result.read(),'html.parser')

table = soup.find_all(lambda tag: tag.name=='table')
for t in table:
    rows = t.find_all(lambda tag: tag.name=='tr')
    for r in rows:
        # there are also the A-Z headers -> check length
        # there are also empty headers -> check isspace()
        headers = r.find_all(lambda tag: tag.name=='h3' and tag.text.isspace()==False and len(tag.text.strip()) > 2)
        for h in headers:
            universities.append(h.text)