使用Python LXML XPath收集数据

Question

I am trying to parse a website for

blahblahblah 
<a  href="THIS IS WHAT I WANT" title="NOT THIS">I DONT CARE ABOUT THIS EITHER</a>
blahblahblah 

(there are many of these, and I want all of them in some tokenized form). The problem is that "a href" actually has two spaces, not just one (there are some that are "a href" with one space that I do NOT want to retrieve), so using tree.xpath('//a/@href') doesn't quite work. Does anyone have any suggestions on what to do?

Thanks!

Answer 1

不了解LXML，但是您可以肯定地使用BeautifulSoup，在页面上找到所有<a> ，然后创建for循环，在该循环中，您将检查<a href=...>与您的正则表达式模式匹配。匹配，而不是报废网址。

Answer 2

This code works as expected :

from lxml import etree

file = "file:///path/to/file.html" # can be a http URL too
doc = etree.parse(file)

print doc.xpath('//a/@href')[0]

Edit : it's not possible AFAIK to do what you want with lxml .

You can use a regex instead.

Answer 3

"(there are some that are "a href" with one space that I do NOT want to retrieve)"

I think this means that you only want to locate elements where there is more than one space between the a and the href. XML allows any amount of whitespace between tag name and attribute (spaces, tabs, new lines are all allowed). The whitespace is discarded by the time the text is parsed and the document tree is created. LXML and XPATH are working with Node objects in the Document tree, not the original text that was parsed to make the tree.

One option is to use regular expressions to find the text you want. But really, since this is perfectly valid XML/HTML, why bother to remove a few spaces?

Answer 4

Use an xpath expression to find all the nodes, then iterate through all those nodes looking for a match, you can obtain a string representation of the node with :

etree.tostring(node)

For futher reference : http://lxml.de/tutorial.html#elements-carry-attributes-as-a-dict