尝试使用 lxml.html 从网站的某个部分获取文本

Question

I have some current Python code that is supposed to get the HTML from a certain part of a website, using the xpath of where the HTML tag is located.

def wordorigins(word):
    pageopen = lxml.html.fromstring("http://www.merriam-webster.com/dictionary/" + str(word))
    pbody = pageopen.xpath("/html/body/div[1]/div/div[4]/div/div[1]/main/article/div[5]/div[3]/div[1]/div/p[1]")
    etybody = lxml.html.fromstring(pbody)
    etytxt = etybody.xpath('text()')
    etytxt = etytxt.replace("<em>", "")
    etytxt = etytxt.replace("</em>", "")
    return etytxt

This code returns this error about expecting a string or a buffer:

Traceback (most recent call last):
  File "mott.py", line 47, in <module>
    print wordorigins(x)
  File "mott.py", line 30, in wordorigins
    etybody = lxml.html.fromstring(pbody)
  File "/usr/lib/python2.7/site-packages/lxml/html/__init__.py", line 866, in fromstring
    is_full_html = _looks_like_full_html_unicode(html)
TypeError: expected string or buffer

Thoughts?

Answer 1

xpath() method returns a list of results , fromstring() expects a string.

But, you don't need to reparse the part of the document. Just use what you've already found:

def wordorigins(word):
    pageopen = lxml.html.fromstring("http://www.merriam-webster.com/dictionary/" + str(word))
    pbody = pageopen.xpath("/html/body/div[1]/div/div[4]/div/div[1]/main/article/div[5]/div[3]/div[1]/div/p[1]")[0]
    etytxt = pbody.text_content()
    etytxt = etytxt.replace("<em>", "")
    etytxt = etytxt.replace("</em>", "")
    return etytxt

Note that I'm using text_content() method instead of the xpath("text()") .

Answer 2

As mentioned in @alecxe 's answer, the xpath() method returns list of matched elements in this case, hence the error when you tried to pass the list to lxml.html.fromstring() . Another thing to note is, that neither XPath's text() function nor lxml 's text_content() method would ever return string containing tag such as <em></em> . They automatically strips tags if any, so the two replace() lines are not needed. You can simply use text_content() or XPath's string() function (instead of text() ) :

......
# either of the following lines should be enough
etytxt = pbody[0].xpath('string()')
etytxt = pbody[0].text_content()