如何通过 xpath 从 htmlresponse 获取所有 br 标签的文本？

Question

I use scrapy to get a object with type htmlresponse for example:

<p class="post">
        line1<br />
        line2<br />
        line3<br />
</p>
<p class="post">
        line4<br />
        line5<br />
        line6<br />
</p>

and I want get all text between p,but I try:

response.xpath('//p[@class="post"]/text()').extract()

but the result have 6 lines from line1 to line6,and I want to get all text between ap tag,for example: #first p line1 line2 line3 #second p line4 line5 line6 How can I do it?

Answer 1

如果您使用的是xslt 2.0 ，则可以使用string-join功能。

string-join(//p[@class="post"]/text())

Answer 2

you can use BeautifulSoup too for parse html(pip install BeautifulSoup4)

from bs4 import BeautifulSoup

html = """
<p class="post">
        line1<br />
        line2<br />
        line3<br />
</p>
<p class="post">
        line4<br />
        line5<br />
        line6<br />
</p>
"""
soup = BeautifulSoup(html, "html.parser")
p=soup.find_all('p')
for x in p:
  print(x.text)

result :

line1
line2
line3


line4
line5
line6

Answer 3

Why do you have to use XPath? BS4 is a good solution. So is SimplifiedDoc

import requests
from simplified_scrapy.simplified_doc import SimplifiedDoc 
html='''<p class="post">
        line1<br />
        line2<br />
        line3<br />
</p>
<p class="post">
        line4<br />
        line5<br />
        line6<br />
</p>
'''
doc = SimplifiedDoc(html)
p = doc.getElementsByClass('post')
for x in p:
  print (x.html)
  print (doc.removeHtml(x.html,' '))

result:

line1<br />line2<br />line3<br />
line1 line2 line3
line4<br />line5<br />line6<br />
line4 line5 line6

Answer 4

Simply write response.css('p ::text').extract() . You can also use the class attribute response.css('.post ::text').extract()

Answer 5

With scrapy selectors you need something like this:

result = [[line.strip("\n ") for line in p_tag.css("*::text").extract() if line.strip("\n ")]
          for p_tag in response.css("p.post")]

#result= [['line1', 'line2', 'line3'], ['line4', 'line5', 'line6']]