BeautifulSoup：找到所有内容在<p>两个之间的标签</p><h2>标签</h2>

Question

我有一个网页要解析，来源如下：

<div class=WordSection1>

<p class=MsoTitle><a name="_nvfbzzqeywr7"></a><span lang=EN>Geometry</span></p>
<h2><a name="_99n9742wmg4y"></a><span lang=EN>Algebraic Geometry </span></h2>

<p class=MsoNormal><span lang=EN>It is a type of geometry which deals with
zeros of multivariate polynomial. It consists of linear and polynomial
algebraic equations used to solve the sets of zeros. The uses of this type <span
class=GramE>consists</span> of Cryptography, String theory, etc.</span></p>

<h2><a name="_64xtqrllvykm"></a><span lang=EN>Discrete Geometry</span></h2>

<p class=MsoNormal><span lang=EN>It is a type of Geometry, mainly concerned
with the relative position of simple geometric objects, such as points, lines,
Triangles, Circles, etc.</span></p>

<h2><a name="_mdul98ybu9wv"></a><span lang=EN>Differential Geometry</span></h2>

<p class=MsoNormal><span lang=EN>It uses techniques of algebra and calculus for
problem solving. The different problem involves general relativity in physics <span
class=SpellE><span class=GramE>etc</span></span><span class=GramE>,.</span></span></p>

</div>

我想以列表的形式解析内部的数据，并在另一个列表中解析相应的标签，以便稍后以 python 字典的形式 map 它。我什至不得不忽略其他标签，如 h1 和 h3。

预期成绩：

headers = ['Algebraic Geometry','Discrete Geometry','Differential Geometry']
content = ['It is a type of geometry which deals with
zeros of multivariate polynomial. It consists of linear and polynomial
algebraic equations used to solve the sets of zeros. The uses of this type consists of Cryptography, String theory, etc.','It is a type of Geometry, mainly concerned
with the relative position of simple geometric objects, such as points, lines,
Triangles, Circles, etc.','It uses techniques of algebra and calculus for
problem solving. The different problem involves general relativity in physics etc,.']

我能够得到所有的标题。但是为了得到

标题内的标签，我无法得到结果。 这是我尝试过的：

content = []

# find the node with id of "WordSection1"
mark = soup.find_all(class_="WordSection1")
print(mark)

# walk through the siblings of the parent (H2) node 
# until we reach the next H2 node
for elt in mark.parent.nextSiblingGenerator():
    if elt.name == "h2":
        break
    if hasattr(elt, "text"):
        content.append(elt.text)

请帮助我。

Answer 1

这应该做你需要的

# read html code snippet
soup = BeautifulSoup(html, 'html.parser')

# find the node with id of "WordSection1"
mark = soup.find(class_="WordSection1")

# initialize container objects
headers = []
content = []

# get all h2 and p(with class MsoNormal) tags
find_headers = mark.find_all_next('h2')
find_content = mark.find_all_next('p', {'class': 'MsoNormal'})

# get text of tags
for head, cont in zip(find_headers, find_content):
   headers.append(head.text)
   content.append(cont.text)

# output lists
print(f"Headers = {headers} '\n\nContent = {content}")```

Answer 2

我尝试使用 Xpath 在Web 抓取语言中表达这一点

GOTO targeturl.com
EXTRACT {'header': '//h2/preceding-sibling::p[@class="MsoNormal"]', 
         'desc': '//p[@class="MsoNormal"]/following-sibling::h2'}

这应该产生一组标题和描述。 但是，由于父项中不包含两个字段，因此如果其中一个条目缺少描述，您将不会获得一对一的匹配。

Answer 3

另一种解决方案。 如果我们只是按照您的示例代码，FireM 的答案是正确的。 您应该发布整个 HTML 结构。

from simplified_scrapy import SimplifiedDoc,req,utils
html = 'your html'
doc = SimplifiedDoc(html)
h2s = doc.selects('div.WordSection1>h2')
ps = h2s.next.text
h2s = h2s.text
print (h2s)
print (ps)

结果：

['Algebraic Geometry', 'Discrete Geometry', 'Differential Geometry']
['It is a type of geometry which deals with zeros of multivariate polynomial. It consists of linear and polynomial algebraic equations used to solve the sets of zeros. The uses of this type consists of Cryptography, String theory, etc.', 'It is a type of Geometry, mainly concerned with the relative position of simple geometric objects, such as points, lines, Triangles, Circles, etc.', 'It uses techniques of algebra and calculus for problem solving. The different problem involves general relativity in physics etc,.']