BeautifulSoup - 使用过滤文本提取特定元素

Question

I'm relatively new to webpage crawling, and I need to extract a particular element out of it, in this case 'Research Project – Cooperative Agreements' right after the hyperlink in the data column.

I've been searching for 'Search_Type=Activity' in the hyperlink using the following code:

for elem in soup(href=lambda href: href and "Search_Type=Activity" in href):
    print (elem.parent)

Because I'm crawling a bunch of NIH grant pages, and I need content of "Activity Code" and they all appear right after the hyperlink with terms 'Search_Type=Sctivity' in it.

So here is the HTML content that I've narrowed down using the code:

<div class="col-md-8 datacolumn"> <a href="//grants.nih.gov/grants/funding/ac_search_results.htm?text_curr=u01&amp;Search.x=0&amp;Search.y=0&amp;Search_Type=Activity">U01</a> Research Project – Cooperative Agreements
        <!--</div>
                </div> end row -->
<!-- If it is not the first row we close the previous row div tags -->
</div>

FYI, The original page used is just an NIH grant here .
Could someone point out what that element is and how to get it out from there?

Answer 1

Try:

import requests
from bs4 import BeautifulSoup

url = "https://grants.nih.gov/grants/guide/rfa-files/RFA-DK-19-501.html"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

name = (
    soup.select_one('[href*="Search_Type=Activity"]')
    .find_next_sibling(text=True)
    .strip()
)
print(name)

Prints:

Research Project – Cooperative Agreements