关于Rap Genius w / Python的Web Scraping Rap歌词

Question

I am somewhat of a coding novice, and I have been trying to scrape Andre 3000's lyrics off Rap genius, http://genius.com/artists/Andre-3000 , by using Beautiful Soup (A Python library for pulling data out of HTML and XML files). My end goal is to have the data in a string format. Here is what I have so far:

from bs4 import BeautifulSoup
from urllib2 import urlopen

artist_url = "http://rapgenius.com/artists/Andre-3000"

def get_song_links(url):
    html = urlopen(url).read()
    # print html 
    soup = BeautifulSoup(html, "lxml")
    container = soup.find("div", "container")
    song_links = [BASE_URL + dd.a["href"] for dd in container.findAll("dd")]

    print song_links

get_song_links(artist_url)
for link in soup.find_all('a'):
    print(link.get('href'))

So I need help with the rest of the code. How do I get his lyrics into string format? and then how do I use the Natural Language Toolkit (NLTK) to token the sentences and words.

Answer 1

Here's an example, how to grab all of the song links on the page, follow them and get the song lyrics:

from urlparse import urljoin
from bs4 import BeautifulSoup
import requests


BASE_URL = "http://genius.com"
artist_url = "http://genius.com/artists/Andre-3000/"

response = requests.get(artist_url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36'})

soup = BeautifulSoup(response.text, "lxml")
for song_link in soup.select('ul.song_list > li > a'):
    link = urljoin(BASE_URL, song_link['href'])
    response = requests.get(link)
    soup = BeautifulSoup(response.text)
    lyrics = soup.find('div', class_='lyrics').text.strip()

    # tokenize `lyrics` with nltk

Note that requests module is used here. Also note that User-Agent header is required since the site returns 403 - Forbidden without it.

Answer 2

First, for each link you will need to download that page and parse it with BeautifulSoup. Then look for a distinguishing attribute on that page that separates lyrics from other page content. I found <a data-editorial-state="accepted" data-classification="accepted" data-group="0"> to be helpful. Then run a .find_all on the lyrics page content to get all lyric lines. For each line you can call .get_text() to get the text from each lyrics line.

As for NLTK, once it is installed you can import it and parse sentences like so:

from nltk.tokenize import word_tokenize, sent_tokenize
words = [word_tokenize(t) for t in sent_tokenize(lyric_text)]

This will give you a list of all words in each sentence.

Answer 3

GitHub / jashanj0tsingh / LyricsScraper.py provides basic scraping of lyrics off genius.com into a text file where each row represents a song. It takes the artist's name as an input. The generated text file then can be easily fed to your custom nltk or the general parser to do stuff you want.

The code is below:

# A simple script to scrape lyrics from the genius.com based on atrtist name.

import re
import requests
import time
import codecs

from bs4 import BeautifulSoup
from selenium import webdriver

mybrowser = webdriver.Chrome("path\to\chromedriver\binary") # Browser and path to Web driver you wish to automate your tests cases.

user_input = input("Enter Artist Name = ").replace(" ","+") # User_Input = Artist Name
base_url = "https://genius.com/search?q="+user_input # Append User_Input to search query
mybrowser.get(base_url) # Open in browser

t_sec = time.time() + 60*20 # seconds*minutes
while(time.time()<t_sec): # Reach the bottom of the page as per time for now TODO: Better condition to check end of page.
    mybrowser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    html = mybrowser.page_source
    soup = BeautifulSoup(html, "html.parser")
    time.sleep(5)

pattern = re.compile("[\S]+-lyrics$") # Filter http links that end with "lyrics".
pattern2 = re.compile("\[(.*?)\]") # Remove unnecessary text from the lyrics such as [Intro], [Chorus] etc..

with codecs.open('lyrics.txt','a','utf-8-sig') as myfile:
    for link in soup.find_all('a',href=True):
            if pattern.match(link['href']):
                f = requests.get(link['href'])
                lyricsoup = BeautifulSoup(f.content,"html.parser")
                #lyrics = lyricsoup.find("lyrics").get_text().replace("\n","") # Each song in one line.
                lyrics = lyricsoup.find("lyrics").get_text() # Line by Line
                lyrics = re.sub(pattern2, "", lyrics)
                myfile.write(lyrics+"\n")
mybrowser.close()
myfile.close()

Answer 4

Hope this is still relevant! I'm doing the same thing with Eminem's lyrics, but from lyrics.com. Does it have to be from Rap Genius? I found lyrics.com to be easier to scrape.

To get Andre 3000's just change the code accordingly.

Here's my code; it gets song links and then scrapes those pages to get lyrics and appends the lyrics to a list:

import re
import requests
import nltk
from bs4 import BeautifulSoup

url = 'http://www.lyrics.com/eminem'
r = requests.get(url)
soup = BeautifulSoup(r.content)
gdata = soup.find_all('div',{'class':'row'})

eminemLyrics = []

for item in gdata:
    title = item.find_all('a',{'itemprop':'name'})[0].text
    lyricsdotcom = 'http://www.lyrics.com'
    for link in item('a'):
        try:
            lyriclink = lyricsdotcom+link.get('href')
            req = requests.get(lyriclink)
            lyricsoup = BeautifulSoup(req.content)
            lyricdata = lyricsoup.find_all('div',{'id':re.compile('lyric_space|lyrics')})[0].text
            eminemLyrics.append([title,lyricdata])
            print title
            print lyricdata
            print
        except:
            pass

This will give you the lyrics in a list. To print all titles:

titles = [i[0] for i in eminemLyrics]
print titles

To get a specific song:

titles.index('Cleaning out My Closet')
120

To tokenize the song, plug that value ( 120 ) in:

song = nltk.word_tokenize(eminemLyrics[120][1])
nltk.pos_tag(song)

Answer 5

Even if you can scrape the site, doesn't mean that you should, instead you can use the API from genius , just create the access token from the Genius API site

import lyricsgenius as genius #calling the API
api=genius.Genius('youraccesstokenhere12345678901234567890isreallylongiknow')
artist=api.search_artist('The artist name here')
aux=artist.save_lyrics(format='json', filename='artist.txt',overwrite=True, skip_duplicates=True,verbose=True)#you can change parameters acording to your needs,i dont recommend using this file directly because it saves a lot of data that you might not need and will take more time to clean it

titles=[song['title'] for song in aux['songs']]#in this case for example i just want title and lyrics
lyrics=[song['lyrics'] for song in aux['songs']]
thingstosave=[]
for i in range(0,128):
    thingstosave.append(titles[i])
    thingstosave.append(lyrics[i])
with open("C:/whateverfolder/alllyrics.txt","w") as output:
    output.write(str(thingstosave))