Scrapy - logging to file and stdout simultaneously, with spider names

Question

I\'ve decided to use the Python logging module because the messages generated by Twisted on std error is too long, and I want to INFO level meaningful messages such

Answer 1

As the Scrapy Official Doc said:

Scrapy uses Python’s builtin logging system for event logging.

So you can config your logger just as a normal Python script.

First, you have to import the logging module:

import logging

You can add this line to your spider:

logging.getLogger().addHandler(logging.StreamHandler())

It adds a stream handler to log to console.

After that, you have to config logging file path.

Add a dict named custom_settings which consists of your spider-specified settings:

custom_settings = {
     'LOG_FILE': 'my_log.log',
     'LOG_LEVEL': 'INFO',
     ... # you can add more settings
 }

The whole class looks like:

import logging

class AbcSpider(scrapy.Spider):
    name: str = 'abc_spider'
    start_urls = ['you_url']
    custom_settings = {
         'LOG_FILE': 'my_log.log',
         'LOG_LEVEL': 'INFO',
         ... # you can add more settings
     }
     logging.getLogger().addHandler(logging.StreamHandler())

     def parse(self, response):
        pass