Scrapy-将Excel .csv导入为start_url

Question

So I'm building a scraper that imports a .csv excel file which has one row of ~2,400 websites (each website is in its own column) and using these as the start_url. I keep getting this error saying that I am passing in a list and not a string. I think this may be caused by the fact that my list basically just has one reallllllly long list in it that represents the row. How can I overcome this and basically put each website from my .csv as its own seperate string within the list?

raise TypeError('Request url must be str or unicode, got %s:' % type(url).__name__)
    exceptions.TypeError: Request url must be str or unicode, got list:


import scrapy
from scrapy.selector import HtmlXPathSelector
from scrapy.http import HtmlResponse
from tutorial.items import DanishItem
from scrapy.http import Request
import csv

with open('websites.csv', 'rbU') as csv_file:
  data = csv.reader(csv_file)
  scrapurls = []
  for row in data:
    scrapurls.append(row)

class DanishSpider(scrapy.Spider):
  name = "dmoz"
  allowed_domains = []
  start_urls = scrapurls

  def parse(self, response):
    for sel in response.xpath('//link[@rel="icon" or @rel="shortcut icon"]'):
      item = DanishItem()
      item['website'] = response
      item['favicon'] = sel.xpath('./@href').extract()
      yield item

Thanks!

Joey

Answer 1

Just generating a list for start_urls does not work as it is clearly written in Scrapy documentation .

From documentation:

You start by generating the initial Requests to crawl the first URLs, and specify a callback function to be called with the response downloaded from those requests.

The first requests to perform are obtained by calling the start_requests() method which (by default) generates Request for the URLs specified in the start_urls and the parse method as callback function for the Requests.

I would rather do it in this way:

def get_urls_from_csv():
    with open('websites.csv', 'rbU') as csv_file:
        data = csv.reader(csv_file)
        scrapurls = []
        for row in data:
            scrapurls.append(row)
        return scrapurls


class DanishSpider(scrapy.Spider):

    ...

    def start_requests(self):
        return [scrapy.http.Request(url=start_url) for start_url in get_urls_from_csv()]

Answer 2

Try opening the .csv file inside the class (not outside as you have done before) and append the start_urls. This solution worked for me. Hope this helps :-)

    class DanishSpider(scrapy.Spider):
        name = "dmoz"
        allowed_domains = []
        start_urls = []

        f = open('websites.csv'), 'r')
        for i in f:
        u = i.split('\n')
        start_urls.append(u[0])

Answer 3

  for row in data:
    scrapurls.append(row)

row is a list [column1, column2, ..] So I think you need to extract the columns, and append to your start_urls.

  for row in data:
      # if all the column is the url str
      for column in row:
          scrapurls.append(column)

Answer 4

Try this way also,

filee = open("filename.csv","r+")

# Removing the \n 'new line' from the url

r=[i for i in filee]
start_urls=[r[j].replace('\n','') for j in range(len(r))]

Answer 5

I find the following useful when in need:

import csv
import scrapy

class DanishSpider(scrapy.Spider):
    name = "rei"
    with open("output.csv","r") as f:
        reader = csv.DictReader(f)
        start_urls = [item['Link'] for item in reader]

    def parse(self, response):
        yield {"link":response.url}