抓取音乐网站以获取歌词

Question

I want to crawl a lyrics website: http://mp3.zing.vn/bai-hat/Vi-Anh-La-Soai-Ca-Dam-Vinh-Hung/ZW78EUE8.html to get the song's name, artist, genre and lyrics. Then I write the following code and save it as mp3_spider.py

import scrapy

class MP3Spider(scrapy.Spider):
    name = "mp3"
    start_urls = ['http://mp3.zing.vn/bai-hat/Vi-Anh-La-Soai-Ca-Dam-Vinh-Hung/ZW78EUE8.html']

def parse(self, response):
    yield
    {
        'song': response.css('.txt-primary h1::text').extract()[0],
        'artist': response.css('.artist-track-log a::text').extract()[0],
        'genre': response.css('.genre-track-log::text').extract()[0],
        'lyrics': response.css('.fn-content::text').extract()[0]
    }

I ran it in command line:

scrapy runspider mp3_spider.py -o mp3.json

but it returns nothing. Can anyone show me how to make it works? Thank you very much for your help.

Answer 1

Your class, MP3Spider , doesn't actually do anything because parse is a standalone function there. If you indent parse to match the indent like this, it'll at least run.

class MP3Spider(scrapy.Spider):
    name = "mp3"
    start_urls = ['http://mp3.zing.vn/bai-hat/Vi-Anh-La-Soai-Ca-Dam-Vinh-Hung/ZW78EUE8.html']

    def parse(self, response):
        yield
        {
            'song': response.css('.txt-primary h1::text').extract()[0],
            'artist': response.css('.artist-track-log a::text').extract()[0],
            'genre': response.css('.genre-track-log::text').extract()[0],
            'lyrics': response.css('.fn-content::text').extract()[0]
        }

Answer 2

I took the liberty of recreating the scenario and aside from the previous poster answer.... Indentation levels are extremely important on how Python interprets your code: what to do, not to do next or before. In addition:

 def parse(self, response):
        yield
        {
            'song': response.css('.txt-primary h1::text').extract()[0],#here
            'artist': response.css('.artist-track-log a::text').extract()[0]#here,
            'genre': response.css('.genre-track-log::text').extract()[0],#here
            'lyrics': response.css('.fn-content::text').extract()[0]#here
        }

May I ask how do you come up with you extract values? I assume you might not use "scrappy shell 'your.com... I assume because by inserting what you had it would tell you that the ranges.. = [0] .... does not exist, at least for the path selected.

I took the liberty in fixing you code up.. But since I don't know Vietnamese , you might have to mess around with some of the regex.

Tips:

1. Though not necessarily important, when you are scraping content that has paragraphs, its best to go with using itemized selections, tend to make the grouping of large bodies of text easier and less regex needed in my experience.

2. Get used to using Scrapy shell function and do all your path selections in there. This will sqave you much time if you use it AND make a habit of very first thing type view(response). Dynamically loaded pages or web pages that block Scrapy's default agent header just won't be easy as just a regular page (still pretty easy, there are always ways around that).

_author_ = 'Tô Vạn Hưng'
__credits__ = 'scriptso' #just helping a brother from the far east
import scrapy

class MP3Spider(scrapy.Spider):
    name = "mp3"
    start_urls = ['http://mp3.zing.vn/bai-hat/Vi-Anh-La-Soai-Ca-Dam-Vinh-Hung/ZW78EUE8.html']

    def parse(self, response):
        yield
        {
            'song': response.css('s.fn-name::text').extract(),
            'artist': response.css('.inline h2::text').extract_first(),
            'genre': response.xpath("//div[@class='inline']/h2/a[contains(font.font,'')]//text()").re('[^\n].*\w')[2:],
            'lyrics': response.css('.fn-wlyrics.fn-content::text').re('[^\n].*\w+'),
        }