从 SQLite 加载 pickle object 时出错

Question

我将 class object 作为泡菜存储在 SQLite 数据库中。

下面是文件pickle.py的代码

sqlite3.register_converter("pickle", pickle.loads)
sqlite3.register_adapter(list, pickle.dumps)
sqlite3.register_adapter(set, pickle.dumps)

class F:

    a = None
    b = None

    def __init__(self) -> None:
        pass
df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
f = F()
f.a = df
f.b = df.columns

data = pickle.dumps(f, protocol=pickle.HIGHEST_PROTOCOL)

sqliteConnection = sqlite3.connect('SQLite_Python.db')
cursor = sqliteConnection.cursor()
print("Successfully Connected to SQLite")
DATA = sqlite3.Binary(data)
sqlite_insert_query = f"""INSERT INTO PICKLES1 (INTEGRATION_NAME, DATA) VALUES ('James',?)"""

resp = cursor.execute(sqlite_insert_query,(DATA,))
sqliteConnection.commit()

之后，我试图从数据库中获取泡菜。 pickle 存储在 pickle 数据类型列中，我之前在 SQLite 的文件retrieve_pickle.py中注册了该列。

cur = conn.cursor()
    cur.execute("SELECT DATA FROM PICKLES1 where INTEGRATION_NAME='James'")
    df = None
    rows = cur.fetchall()
    for r in rows[0]:
        print(type(r)) #prints <class 'bytes'>
        df = pickle.loads(r)

但这给了我一个错误

  File "/Users/ETC/Work/pickle_work/picklertry.py", line 34, in select_all_tasks
    df = pickle.loads(r)
AttributeError: Can't get attribute 'F' on <module '__main__' from '/Users/rusab1/Work/pickle_work/picklertry.py'>

在将 pickle.loads 注册为 pickle 数据类型后，我试图将 class object 存储在 sqlite 的 pickle 列中。 我成功地保留了 object 并能够从数据库中检索它，但是当我尝试将其加载回来以便我可以访问该事物和属性时，它给我一个错误。

Answer 1

Pickling 要求您导入 pickle 中的实际模块。 我必须将 F 导入到我正在加载泡菜的第二个文件中。

Answer 2

您没有指定数据库的实际创建方式。

首先，我会在连接语句上使用额外的 arguments: detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES 。

然后我将声明“DATA”列的类型为pickle 。 然后，当您输入 select 时，“DATA”列的二进制文件将自动解封：

#!/usr/bin/env python3

from sqlite_common import *
import pandas as pd
import pickle

sqlite3.register_converter("pickle", pickle.loads)
sqlite3.register_adapter(list, pickle.dumps)
sqlite3.register_adapter(set, pickle.dumps)

class F:

    a = None
    b = None

    def __init__(self) -> None:
        pass
df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
f = F()
f.a = df
f.b = df.columns

data = pickle.dumps(f, protocol=pickle.HIGHEST_PROTOCOL)

sqliteConnection = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
print("Successfully Connected to SQLite")
sqliteConnection.execute("""
    create table PICKLES1 (
        "INTEGRATION_NAME" text unique not null,
        "DATA" pickle not null
    )
""")

cursor = sqliteConnection.cursor()
# This is unnecessary:
#DATA = sqlite3.Binary(data)
sqlite_insert_query = f"""INSERT INTO PICKLES1 (INTEGRATION_NAME, DATA) VALUES ('James',?)"""

resp = cursor.execute(sqlite_insert_query,(data,))
sqliteConnection.commit()

cur = sqliteConnection.cursor()
cur.execute("SELECT DATA FROM PICKLES1 where INTEGRATION_NAME='James'")
rows = cur.fetchall()
for r in rows: # Note this change
    f = r[0]
    print(f.a)
    print(f.b)

印刷：

Successfully Connected to SQLite
   col1  col2
0     1     3
1     2     4
Index(['col1', 'col2'], dtype='object')

即使使用当前定义“DATA”列的方式，如果按照我建议的方式连接，您也可以强制将数据解释为 pickle 序列化数据，并使用以下语句自动反序列化：

cur.execute("SELECT DATA [pickle] FROM PICKLES1 where INTEGRATION_NAME='James'")

您甚至可以为 class F定义适配器并取消对pickle.dumps的显式调用：

#!/usr/bin/env python3

from sqlite_common import *
import pandas as pd
import pickle

class F:

    a = None
    b = None

    def __init__(self) -> None:
        pass

sqlite3.register_converter("pickle", pickle.loads)
sqlite3.register_adapter(list, pickle.dumps)
sqlite3.register_adapter(set, pickle.dumps)
sqlite3.register_adapter(F, pickle.dumps)

df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
f = F()
f.a = df
f.b = df.columns

sqliteConnection = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
print("Successfully Connected to SQLite")
sqliteConnection.execute("""
    create table PICKLES1 (
        "INTEGRATION_NAME" text unique not null,
        "DATA" pickle not null
    )
""")

cursor = sqliteConnection.cursor()
sqlite_insert_query = f"""INSERT INTO PICKLES1 (INTEGRATION_NAME, DATA) VALUES ('James',?)"""

resp = cursor.execute(sqlite_insert_query,(f,))
sqliteConnection.commit()

cur = sqliteConnection.cursor()
cur.execute("SELECT DATA FROM PICKLES1 where INTEGRATION_NAME='James'")
rows = cur.fetchall()
for r in rows:
    f = r[0]
    print(f.a)
    print(f.b)