从网页中提取特定的页面链接

Question

I am using BeautifulSoup to extract all the links from this page: http://kern.humdrum.org/search?s=t&keyword=Haydn

I am getting all these links this way:

# -*- coding: utf-8 -*-

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'http://kern.humdrum.org/search?s=t&keyword=Haydn'

#opening up connecting, grabbing the page
uClient = uReq(my_url)

# put all the content in a variable
page_html = uClient.read()

#close the internet connection
uClient.close()

#It does my HTML parser
page_soup = soup(page_html, "html.parser")

# Grab all of the links
containers = page_soup.findAll('a', href=True)
#print(type(containers))

for container in containers:
    link = container
    #start_index = link.index('href="') 
    print(link)
    print("---")
    #print(start_index)

part of my output is:

Notice that it is returning several links but I really want all the ones with >Someting. (For example, ">Allegro" and "Allegro vivace" and so forth).

I am having a hard time getting the following type of output (example of the image): "Allegro - http://kern.ccarh.org/cgi-bin/ksdata?location=users/craig/classical/beethoven/piano/sonata&file=sonata01-1.krn&format=info "

In other words, at this point, I have a bunch of anchor tags (+- 1000). From all these tags there are a bunch that are just "trash" and +- 350 of tags that I would like to extract. All these tags look almost the same but the only difference is that the tags that I need have a "> Somebody's name<\\a>" at the end. I would like to exctract only the link of all the anchor tags with this characteristic.

Answer 1

From what I can see in the image the ones with info have an href attribute containing format="info" so you could use an attribute=value CSS selector of [href*=format="info"] , where the * indicates contains ; the attribute value contains the substring after the first equals.

import bs4 , requests

res = requests.get("http://kern.humdrum.org/search?s=t&keyword=Haydn")
soup = bs4.BeautifulSoup(res.text,"html.parser")
for link in soup.select('[href*=format="info"]'):
    print(link.getText(), link['href'])

Answer 2

The best and easiest way is using text attribute when printing the link. like this : print link.text

Answer 3

Assuming you already have a list of the substrings you need to search for, you can do something like:

for link in containers:
    text = link.get_text().lower()
    if any(text.endswith(substr) for substr in substring_list):
        print(link)
        print('---')

Answer 4

you want to extract link with specified anchor text?

for container in containers:
    link = container
    # match exact
    #if 'Allegro di molto' == link.text:
    if 'Allegro' in link.text: # contain
        print(link)
        print("---")