需要使用漂亮的汤从锚标签中提取链接和文本

Question

我正在努力使用漂亮的汤从锚标签中提取链接和文本

下面的代码是我必须从锚标签中提取数据的地方，锚标签是链接和文本

 <a href="https://indianexpress.com/article/cities/mumbai/mumbai-vaccination-figures-surge-in-private-hospitals-stagnate-in-government-centres-7338768/">Mumbai: Vaccination figures surge in private hospitals, stagnate in government centres</a> <a href="https://indianexpress.com/article/cities/chennai/chennai-martial-arts-instructor-arrested-following-allegations-of-sexual-assault-7339259/">Chennai: Martial arts instructor arrested following allegations of sexual assault</a> <a href="https://indianexpress.com/article/explained/mumbai-metro-lines-2a-and-7-here-is-everything-you-need-to-know-7338637/">Mumbai Metro lines 2A and 7: Here is everything you need to know</a>

**Python code to extract the content from the above code.**
@app.get('/indian_express', response_class=HTMLResponse)
async def dna_india(request: Request):
    print("1111111111111111")
    dict={}
    URL="https://indianexpress.com/latest-news/"
    page=requests.get(URL)
    soup=BS(page.content, 'html.parser')
    results=soup.find_all('div', class_="nation")
    for results_element in results:
        results_element_1 = soup.find_all('div', class_="title")
        for results_element_2 in results_element_1:
            for results_element_3 in results_element_2:
                print(results_element_3)  **The above printed html code is because of this print**
                print(" ")
                link_element=results_element_3.find_all('a', class_="title", href=True) **I am getting empty [] when i try to print here **
                # print(link_element)
                # title_elem = results_element_3.find('a')['href']
                # link_element=results_element_3.find('a').contents[0]
                # print(title_elem)
                # print(link_element)
                # for index,(title,link) in enumerate(zip(title_elem, link_element)):
                #     dict[str(title.text)]=str(link['href'])
    json_compatible_item_data = jsonable_encoder(dict)
    return templates.TemplateResponse("display.html", {"request":request, "json_data":json_compatible_item_data})


@app.get('/deccan_chronicle', response_class=HTMLResponse)
async def deccan_chronicle(request: Request):
    dict={}
    URL="https://www.news18.com/india/"
    page=requests.get(URL)
    soup=BS(page.content, 'html.parser')
    main_div = soup.find("div", class_="blog-list")
    for i in main_div:
        #link_data = i.find("div", class_="blog-list-blog").find("a")
        link_data=i.find("div",class_="blog-list-blog").find("a")
        text_data = link_data.text
        dict[str(text_data)] = str(link_data.attrs['href'])
    json_compatible_item_data = jsonable_encoder(dict)
    return templates.TemplateResponse("display.html", {"request":request, "json_data":json_compatible_item_data})

请帮我解决这个代码

Answer 1

您可以找到包含所有新闻记录的main_div标签，您可以在其中找到定义了所有数据的articles ，并且可以使用找到包含标题以及相同标题a正确标签来提取articles标题的herf ！

import requests
from bs4 import BeautifulSoup

res=requests.get("https://indianexpress.com/latest-news/")
soup=BeautifulSoup(res.text,"html.parser")

main_div=soup.find("div",class_="nation")
articles=main_div.find_all("div",class_="articles")
for i in articles:
    href=i.find("div",class_="title").find("a")
    print(href.attrs['href'])
    text_data=href.text
    print(text_data)

Output：

https://indianexpress.com/article/business/banking-and-finance/banks-cant-cite-2018-rbi-circular-to-caution-clients-on-virtual-currencies-7338628/
Banks can’t cite 2018 RBI circular to caution clients on virtual currencies
https://indianexpress.com/article/india/supreme-court-stays-delhi-high-court-order-on-levy-of-igst-on-imported-oxygen-concentrators-for-personal-use-7339478/
Supreme Court stays Delhi High Court order on levy of IGST on imported oxygen concentrators for personal use
...

第二种方法

不要那么复杂，只需观察标签它们包含什么数据，就像我找到了主标签main_div然后 go 用于包含文本和链接的标签，您可以在h4标签中找到它并对其进行迭代！

from bs4 import BeautifulSoup
import requests
res=requests.get("https://www.news18.com/india/")

soup=BeautifulSoup(res.text,"html.parser")
main_div = soup.find("div", class_="blog-list")
data=main_div.find_all("h4")
for i in data:
    print(i.find("a")['href'])
    print(i.find("a").text)

output：

https://www.news18.com/news/india/2-killed-six-injured-after-portion-of-two-storey-building-collapses-in-varanasi-pm-assures-help-3799610.html
2 Killed, Six Injured After Portion of Two-Storey Building Collapses in Varanasi; PM Assures Help
https://www.news18.com/news/india/dont-compel-citizens-to-move-courts-again-again-follow-national-litigation-policy-hc-tells-centre-3799598.html
Don't Compel Citizens to Move Courts Again & Again, Follow National Litigation Policy, HC Tells Centre

...