如何使用ASCII字符查询Unicode数据库

Question

我目前在我的postgresql数据库上运行一个查询，该查询忽略德语字符-变音符号。 但是，我不想放弃这些字符，而是希望在查询的输出中使用德语字符或至少它们的等效字符（例如ä= ae）。 运行Python 2.7.12

当我将编码对象更改为replace或xmlcharrefreplace ，出现以下错误：

psycopg2.ProgrammingError: syntax error at or near "?"
LINE 1: ?SELECT

代码段：

# -*- coding: utf-8 -*-

    connection_str = r'postgresql://' + user + ':' + password + '@' + host + '/' + database

    def query_db(conn, sql):
        with conn.cursor() as curs:
            curs.execute(sql)
            rows = curs.fetchall()

        print("fetched %s rows from db" % len(rows))

        return rows

    with psycopg2.connect(connection_str) as conn:
        for filename in files:
            # Read SQL
            sql = u""

            f = codecs.open(os.path.join(SQL_LOC, filename), "r", "utf-8")

            for line in f:
                sql += line.encode('ascii', 'replace').replace('\r\n', ' ')

            rows = query_db(conn, f)

如何将查询作为带有德语字符的unicode对象传递？ 我也尝试将查询解码为utf-8但随后出现以下错误：

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

Answer 1

这是获得其编码等效项的解决方案。 您稍后可以重新编码，查询不会产生错误：

SELECT convert_from(BYTEA 'foo ᚠ bar'::bytea, 'latin-1');
+----------------+
| convert_from   |
|----------------|
| foo á<U+009A>  bar                |
+----------------+
SELECT 1
Time: 0.011s

Answer 2

您只需要conn.set_client_encoding("utf-8") ，然后即可执行unicode字符串conn.set_client_encoding("utf-8") ，结果将即时进行编码和解码：

$ cat psycopg2-unicode.py
import sys
import os
import psycopg2
import csv

with psycopg2.connect("") as conn:
    conn.set_client_encoding("utf-8")
    for filename in sys.argv[1:]:
        file = open(filename, "r", encoding="utf-8")
        sql = file.read()
        with conn.cursor() as cursor:
            cursor.execute(sql)
            try:
                rows = cursor.fetchall()
            except psycopg2.ProgrammingError as err:
                # No results
                continue
            with open(filename+".out", "w", encoding="utf-8", newline="") as outfile:
                csv.writer(outfile, dialect="excel-tab").writerows(rows)

$ cat sql0.sql
create temporary table t(v) as
    select 'The quick brown fox jumps over the lazy dog.'
    union all
    select 'Zwölf große Boxkämpfer jagen Viktor quer über den Sylter Deich.'
    union all
    select 'Любя, съешь щипцы, — вздохнёт мэр, — кайф жгуч.'
    union all
    select 'Mężny bądź, chroń pułk twój i sześć flag.'
;

$ cat sql1.sql
select * from t;

$ python3 psycopg2-unicode.py sql0.sql sql1.sql

$ cat sql1.sql.out 
The quick brown fox jumps over the lazy dog.
Zwölf große Boxkämpfer jagen Viktor quer über den Sylter Deich.
Любя, съешь щипцы, — вздохнёт мэр, — кайф жгуч.
Mężny bądź, chroń pułk twój i sześć flag.

---

该程序的Python2版本稍微复杂一点，因为我们需要告诉驱动程序我们希望将返回值作为unicode对象。 另外，我用于输出的csv模块不支持unicode，因此需要一种解决方法。 这里是：

$ cat psycopg2-unicode2.py
from __future__ import print_function

import sys
import os
import csv
import codecs

import psycopg2
import psycopg2.extensions
psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)

with psycopg2.connect("") as conn:
    conn.set_client_encoding("utf-8")
    for filename in sys.argv[1:]:
        file = codecs.open(filename, "r", encoding="utf-8")
        sql = file.read()
        with conn.cursor() as cursor:
            cursor.execute(sql)
            try:
                rows = cursor.fetchall()
            except psycopg2.ProgrammingError as err:
                # No results from SQL
                continue
            with open(filename+".out", "wb") as outfile:
                for row in rows:
                    row_utf8 = [v.encode('utf-8') for v in row]
                    csv.writer(outfile, dialect="excel-tab").writerow(row_utf8)