如何使用Beautifulsoup-python从div中特定标题的段落元素中的网页元素中提取网页文本

Question

Basically the title. I am trying to pull the paragraph text from the area underneath "genecards summary for name_of_gene gene" from https://www.genecards.org/cgi-bin/carddisp.pl?gene=IL6&keywords=il6 using the IL-6 gene as an example. what I want to pull is would like to pull just "IL6 (Interleukin 6) is a Protein Coding gene. Diseases associated with IL6 include Kaposi Sarcoma and Rheumatoid Arthritis, Systemic Juvenile. Among its related pathways are IL-1 Family Signaling Pathways and Immune response IFN alpha/beta signaling pathway. Gene Ontology (GO) annotations related to this gene include signaling receptor binding and growth factor activity."

I have been trying to use Beautifulsoup 4 with python. The issue I am having specifically is that I just don't know how to specify what text I want to pull from the website.

from bs4 import BeautifulSoup

from urllib.request import Request, urlopen

baseURL = "https://www.genecards.org/cgi-bin/carddisp.pl?gene="
GeneToSearch = input("Gene of Interest: ")`
updatedURL = baseURL + GeneToSearch
print(updatedURL)

req = Request(updatedURL, headers={'User-Agent': 'Mozilla/5.0'})
response = urlopen(req).read()

soup = BeautifulSoup(response, 'lxml')

for tag in soup.find_all(['script', 'style']):
   tag.decompose()
soup.get_text(strip=True)
VALID_TAGS = ['div', 'p']

for tag in soup.findAll('GeneCards Summary for '+ GeneToSearch +    'Gene'):
    if tag.name not in VALID_TAGS:
        tag.replaceWith(tag.renderContents())

print(soup.text)

This just ends up giving me every element from the website.

Answer 1

Try to navigate between tags, something like this:

soup.select('.gc-subsection-header')[1].next_sibling.next_sibling.text

Ref.: Beautiful Soup

Answer 2

Using the latest version of BeautifulSoup you can use a pseudo css selector (:contains) to search for a tag with specific text. You can then navigate to the next p tag and extract corresponding text:

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

baseURL = "https://www.genecards.org/cgi-bin/carddisp.pl?gene="
GeneToSearch = input("Gene of Interest: ")
updatedURL = baseURL + GeneToSearch
print(updatedURL)

req = Request(updatedURL, headers={'User-Agent': 'Mozilla/5.0'})
response = urlopen(req).read()

soup = BeautifulSoup(response, 'lxml')

text_find = 'GeneCards Summary for ' + GeneToSearch + ' Gene'



print(summary)

Outputs:

IL6 (Interleukin 6) is a Protein Coding gene.
Diseases associated with IL6 include Kaposi Sarcoma and Rheumatoid Arthritis, Systemic Juvenile.
Among its related pathways are IL-1 Family Signaling Pathways and Immune response IFN alpha/beta signaling pathway.
Gene Ontology (GO) annotations related to this gene include signaling receptor binding and growth factor activity.