How to actually set utf-8 encoding in SQLAlchemy / Postgres in Python

Question

I'm developing a web application using Flask and a Postgresql database (using SQLAlchemy). It works just fine. Except for a little problem: all text data comes in some encoding that it's not utf-8 (at least I think it's not).

I looked around to see any possible solutions and this is what I've tried:

• Include this line at every Python file in the project:

# -*- coding: utf-8 -*-

• Set the engine like this:

def __init__(self):
        self.engine = sql.create_engine(os.environ['DATABASE_URL'], encoding='utf8')
        self.conn = self.engine.connect()
        self.metadata = sql.MetaData()

And a few variations of these previous items like client_encoding='utf8 instead of encoding='utf8 . I also tried these solutions:

• Encoding error with sqlalchemy and postgresql
• SQLAlchemy result for UTF-8 column is of type 'str', why?
• how to use `charset` and `encoding` in `create_engine` of SQLAlchemy (to create pandas dataframe)?
• python encoding utf-8
• Python encoding unicode utf-8

For example, this is one of my select functions (but the problem happens in every select / insert / update functions):

def get_produtos(self, id):
        produtos = sql.Table('produtos', self.metadata, autoload=True,
                             autoload_with=self.engine).columns
        parceiros = sql.Table('parceiros', self.metadata, autoload=True,
                              autoload_with=self.engine).columns
        q = sql.select([produtos.id, produtos.nome, parceiros.nome_parceiro,
                        produtos.valor, produtos.qtd_desconto,
                        parceiros.id_parceiro])
        q = q.where(produtos.id == id)
        return self.conn.execute(q).fetchall()

The program does not raises any exception but the data just comes with the wrong encoding.

This is how the data is in my database: https://i.imgur.com/8BA1i3N.png

This is how the data shows up in my app: https://i.imgur.com/NZBxFtl.png

Answer 1

As JosefZ said, it's a Mojibake case. This problem in particular occured because some entries were inserted via shell and there was no encoding and led to this inconsistency.