[英]give names to start_urls in scrapy
I am crawling urls from a csv file, and each url has a name. 我正在从一个csv文件中爬网URL,每个URL都有一个名称。 How can I download these urls and save them with their names? 如何下载这些网址并将其保存为名称?
reader = csv.reader(open("source1.csv"))
for Name,Sources1 in reader:
urls.append(Sources1)
class Spider(scrapy.Spider):
name = "test"
start_urls = urls[1:]
def parse(self, response):
filename = **Name** + '.pdf' //how can I get the names I read from the csv file?
Perhaps you want to override the start_requests() method instead of using start_urls? 也许您想覆盖start_requests()方法而不是使用start_urls?
Example: 例:
class MySpider(scrapy.Spider):
name = 'test'
def start_requests(self):
data = read_csv()
for d in data:
yield scrapy.Request(d.url, meta={'name': d.name})
The meta
dict for request will be repassed to the response, so you can later do: 请求的meta
字典将被传递到响应,因此您以后可以执行以下操作:
def parse(self, response):
name = response.meta.get('name')
...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.