How can I create log file with spider name in settings.py dynamically?

Question

I have 20 different spiders and it works scheduled. End of the day, when I check the log file I am getting over 15.000 line log.

My recent log setting in settings.py

from datetime import datetime
now = datetime.today()
now_time = now.strftime("%d.%m.%y")

LOG_FORMAT = '%(levelname)s: %(message)s'
LOG_FILE = 'scrapy-log-'+now_time+'.txt'

But I want to prepare log file depends on spider name like

What I expect in settings.py

from datetime import datetime
now = datetime.today()
now_time = now.strftime("%d.%m.%y")

LOG_FORMAT = '%(levelname)s: %(message)s'
LOG_FILE = spider_name+now_time+'.txt'

But I don't know how to get spider_name dynamically when crawling all spiders.

So the question is how can I use spider_name dynamically in settings.py?

Answer 1

You could try overriding settings upon instantiating the spider. Something like this:

class YourSpider(CrawlSpider):

    start_urls = ['http://website.com']
    allowed_domains = ['www.website.com']

    name = 'spider_name'
    custom_settings = {
        'LOG_FILE' = name+now_time+'.txt',
    }

See section "2. Settings per-spider" in documentation https://docs.scrapy.org/en/latest/topics/settings.html