如何根据scrapy中的日志条目来源有选择地设置log_level？

Question

Is it possible to have the log level for [scrapy.core.engine] and [scrapy.extensions.logstats] to be 'INFO' as well as my custom logger and have everything else set to 'WARNING'? I'd like to do this to remove some of the clutter from my log file.

Thanks in advance!

EDIT:

I tried doing the following as described in this answer:

DEFAULT_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'root': {'level': 'WARNING'},
    'loggers': {
        'twisted': {
            'level': 'ERROR',
        },
        'scrapy.core.engine': {
            'level': 'INFO',
        },
        'scrapy.extensions.logstats': {
            'level': 'INFO',
        },
    }
}

Unfortunately, this doesn't work for some reason. The default 'Warning' level doesn't get applied to all loggers and the specified levels for the loggers above seem to have no effect...

Answer 1

Apparently scrapy uses logging . You could use something like the following:

import logging
for key in logging.Logger.manager.loggerDict:
    if "scrapy.core.engine" in key or "scrapy.extensions.logstat" in key:
        logging.getLogger(key).setLevel(logging.WARNING)
logging.basicConfig(level=logging.INFO)

I haven't tested it with scrapy, but I use it for other packages. Hope it works.