UnicodeEncodeError: 'charmap' codec can't encode character even after formatting logger to use utf-8 encoding

Question

I am getting several of these errors from my logger in python.

UnicodeEncodeError: 'charmap' codec can't encode character X in position Y : character maps to undefined

Looking around on stackoverflow, I see that many people have been able to resolve the issue by telling their logger to use 'utf-8' encoding however for me the error still remains.

This is the function that I am using to create my loggers:

def makeLogger(name):
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)

    formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')

    file_handler = logging.FileHandler(r'C:\Users\lguarro\Documents\Work\SearchEngine_Pure\Log\custom.log', 'a', 'utf-8')
    file_handler.setFormatter(formatter)

    logger.addHandler(file_handler)
    return logger

As you can see I have explicitly told the FileHandler to use utf-8.

Here is an example of how I call my logger from which many errors originate:

self.logger.info("Starting url scrape for company, " + rec["Company"] + " using user agent: " + user_agent)

In particular the error typically comes from the characters inside rec["Company"] as there are a lot of weird company names in my database like lantmännen unibake.

So what exactly am I missing?

Answer 1

My issue was that I was using Scrapy and I was not configuring the root logger correctly and was only setting up the logger for each of my individual spiders (which behaves correctly)

In essence, I needed to add this to my execution script:

root_logger = logging.getLogger()
root_logger.setLevel(logging.WARNING)
handler = logging.FileHandler(r'C:\Users\lguarro\Documents\Work\SearchEngine_Pure\Log\scrapy.log', 'w', 'utf-8')
handler.setFormatter(logging.Formatter('%(asctime)s:%(levelname)s:%(message)s'))
root_logger.addHandler(handler)

For whatever reason, when I was trying to do the same thing using logging.basicConfig, it told me that "encoding" is not a valid attribute so I opted for this method instead.