快速查找链接：正则表达式与lxml

Question

I am trying to build a fast web crawler, and as a result, I need an efficient way to locate all the links on a page. What is the performance comparison between a fast XML/HTML parser like lxml and using regex matching?

Answer 1

The problem here isn't about regex vs lxml. Regex just isn't a solution. How would you restrict the elements from where the links come from? A more real-world example is malformed HTML. How would you extract the contents of the href attribute out of this link?

<A href = /text" data-href='foo>' >Test</a>

lxml parses it just fine, just like Chrome, but good luck getting a regex to work. If you're curious about the actual speed differences, here's a quick test I made.

Setup:

import re
import lxml.html

def test_lxml(html):
    root = lxml.html.fromstring(html)
    #root.make_links_absolute('http://stackoverflow.com/')

    for href in root.xpath('//a/@href'):
        yield href

LINK_REGEX = re.compile(r'href="(.*?)"')

def test_regex(html):
    for href in LINK_REGEX.finditer(html):
        yield href.group(1)

Test HTML:

html = requests.get('http://stackoverflow.com/questions?pagesize=50').text

Results:

In [22]: %timeit list(test_lxml(html))
100 loops, best of 3: 9.05 ms per loop

In [23]: %timeit list(test_regex(html))
1000 loops, best of 3: 582 us per loop

In [24]: len(list(test_lxml(html)))
Out[24]: 412

In [25]: len(list(test_regex(html)))
Out[25]: 416

For comparison, here's how many links Chrome picks out:

> document.querySelectorAll('a[href]').length
413

Also, just for the record, Scrapy is one of the best web scraping frameworks out there and it uses lxml to parse the HTML.

Answer 2

你可以使用pyquery，一个python库，为你带来jquery的功能。