[英]How to save/restore/add elements to Python sets in a Sqlite database?
How to save (and also restore, and add elements to) a set of strings in a Sqlite3 database?如何在 Sqlite3 数据库中保存(以及恢复和添加元素)一组字符串?
This does not work because sets are not JSON-serializable:这不起作用,因为集合不是 JSON 可序列化的:
import sqlite3, json
db = sqlite3.connect(':memory:')
db.execute('CREATE TABLE t(id TEXT, myset TEXT);')
s = {'a', 'b', 'c'}
db.execute("INSERT INTO t VALUES (?, ?);", ('1', json.dumps(s)))
# Error: Object of type set is not JSON serializable
so we can use a list , or a dict with dummy values:所以我们可以使用list或带有虚拟值的 dict :
s = list(s)
# or s = {'a':0, 'b':0, 'c': 0}
db.execute("INSERT INTO t VALUES (?, ?);", ('1', json.dumps(s)))
# RETRIEVE A SET FROM DB
r = db.execute("SELECT myset FROM t WHERE id = '1'").fetchone()
if r is not None:
s = set(json.loads(r[0]))
print(s)
Then adding a string element to a set already in the DB is not very elegant:然后将字符串元素添加到数据库中已有的集合中并不是很优雅:
SELECT
,SELECT
,json.loads
,json.loads
,json.dumps
,json.dumps
JSON化它,UPDATE
UPDATE
Is there a more pythonic way to work with sets in a Sqlite database?在 Sqlite 数据库中是否有更 pythonic 的方式来处理集合?
You can register adapter and converter functions with sqlite that will automatically perform the desired conversions.您可以使用 sqlite 注册适配器和转换器功能,这将自动执行所需的转换。
import json
import sqlite3
def adapt_set(value):
return json.dumps(list(value))
def convert_set(value):
return set(json.loads(value))
sqlite3.register_adapter(set, adapt_set)
sqlite3.register_converter('set_type', convert_set)
Once these functions have been registered, pass detect_types
to the connection factory to tell sqlite how to use them.一旦注册了这些函数,将
detect_types
传递给连接工厂以告诉 sqlite 如何使用它们。
Passing sqlite3.PARSE_DECLTYPE will make the connection use the declared type to look up the adapter/converter.传递sqlite3.PARSE_DECLTYPE将使连接使用声明的类型来查找适配器/转换器。
db = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_DECLTYPES)
# Declare the myset column as type "set_type".
db.execute('CREATE TABLE t(id TEXT, myset set_type);')
db.execute("INSERT INTO t VALUES (?, ?);", ('1', {1, 2, 3}))
r = db.execute("""SELECT myset FROM t WHERE id = '1'""").fetchone()
print(r[0]) # <- r[0] is a set.
Passing sqlite.PARSE_COLNAMES will cause the column name in the cursor description to be searched for the type name enclosed in square brackets.传递sqlite.PARSE_COLNAMES将导致在 cursor 描述中的列名称中搜索方括号中的类型名称。
db = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_COLNAMES)
# The type is not declared in the created table statement.
db.execute('CREATE TABLE t(id TEXT, myset TEXT);')
db.execute("INSERT INTO t VALUES (?, ?);", ('1', {1, 2, 3}))
# Include the type in the column label.
r = db.execute("""SELECT myset "AS myset [set_type]" FROM t WHERE id = '1'""").fetchone()
print(r[0]) <- r[0] is a set
I would register a "set adapter" that converts a set into a byte string by simply taking the string representation of the set and encoding it into a bytes string for storage and a "set converter" that converts our user-defined column type named "set_type" (pick any alternate you wish) from a byte string back into a set by decoding the byte string back into a Unicode string and then applying eval
against it:我会注册一个“集合适配器”,通过简单地获取集合的字符串表示并将其编码为字节字符串进行存储,将集合转换为字节字符串,并注册一个“集合转换器”,将我们名为“的用户定义的列类型转换为”通过将字节字符串解码回 Unicode 字符串然后对其应用
eval
,将字节字符串中的 set_type”(选择您想要的任何替代项)返回到集合中:
import sqlite3
from decimal import Decimal
db = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
def set_adapter(the_set):
return str(the_set).encode('utf-8')
def set_converter(s):
return eval(s.decode('utf-8'))
sqlite3.register_adapter(set, set_adapter)
sqlite3.register_converter('set_type', set_converter)
# Define the columns with type set_type:
db.execute('CREATE TABLE t(id TEXT, myset set_type);')
s = {'a', 'b', 'c', (1, 2, 3), True, False, None, b'abcde', Decimal('12.345')}
# Our adapter will store s as a byte string:
db.execute("INSERT INTO t VALUES (?, ?);", ('1', s))
cursor = db.cursor()
# Our converter will convert column type set_type from bytes to set:
cursor.execute('select myset from t')
row = cursor.fetchone()
s = row[0]
print(type(s), s)
db.close()
Prints:印刷:
<class 'set'> {False, True, 'b', None, (1, 2, 3), Decimal('12.345'), b'abcde', 'a', 'c'}
As you can see, this also handles more datatypes than JSON can after converting a set to a list.如您所见,在将集合转换为列表后,这还可以处理比 JSON 更多的数据类型。 JSON could potentially handle as many data types but only if you write JSON converters (which usually means converting those data types to one of the supported types such as strings).
JSON 可能会处理尽可能多的数据类型,但前提是您编写 JSON 转换器(这通常意味着将这些数据类型转换为一种受支持的类型,例如字符串)。
Defining the adapter and converter to use pickle
as in the answer offered by Jonathan Feenstra will result in a speed improvement but possibly use more storage.如 Jonathan Feenstra 提供的答案那样定义适配器和转换器以使用
pickle
将导致速度提高,但可能会使用更多存储空间。 But I would use the technique outlined above, ie using adapter and converter functions with a special user-defined column type:但我会使用上面概述的技术,即使用具有特殊用户定义列类型的适配器和转换器函数:
import pickle
def set_adapter(the_set):
return pickle.dumps(the_set, pickle.HIGHEST_PROTOCOL)
def set_converter(s):
return pickle.loads(s)
A simpler way to store the set in the SQLite database is to use the BLOB
datatype for the myset
column and serialise it to bytes using pickle.dumps
:将集合存储在 SQLite 数据库中的一种更简单的方法是对
myset
列使用BLOB
数据类型,并使用pickle.dumps
将其序列化为字节:
import sqlite3
import pickle
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE t(id TEXT, myset BLOB)")
s = {"a", "b", "c"}
db.execute(
"INSERT INTO t VALUES (?, ?)",
("1", sqlite3.Binary(pickle.dumps(s, pickle.HIGHEST_PROTOCOL))),
)
r = db.execute("SELECT myset FROM t WHERE id = '1'").fetchone()
if r is not None:
s = pickle.loads(r[0])
print(s)
To add new elements to a set in the database, the serialisation and deserialisation steps are still required, but no more conversion to/from a list or checking for elements that are already present in the set.要将新元素添加到数据库中的集合,仍然需要序列化和反序列化步骤,但不再需要与列表之间的转换或检查集合中已经存在的元素。
Alternatively, you could ensure the uniqueness of the ID-element combination at database-level, for example using a composite primary key:或者,您可以确保 ID 元素组合在数据库级别的唯一性,例如使用复合主键:
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE t(id TEXT, element TEXT, PRIMARY KEY(id, element))")
s = {"a", "b", "c"}
for element in s:
db.execute("INSERT INTO t VALUES(?, ?)", ("1", element))
r = db.execute("SELECT element FROM t WHERE id = '1'").fetchall()
s = set(row[0] for row in r)
print(s)
You can use aiosqlitedict你可以使用aiosqlitedict
Here is what it can do这是它能做什么
- Easy conversion between sqlite table and Python dictionary and vice-versa.
sqlite 表和 Python 字典之间的轻松转换,反之亦然。
- Get values of a certain column in a Python list.
获取 Python 列表中某一列的值。
- Order your list ascending or descending.
按升序或降序排列您的列表。
- Insert any number of columns to your dict.
在您的字典中插入任意数量的列。
We start by connecting our database along with the reference column我们首先连接我们的数据库和参考列
from aiosqlitedict.database import Connect
countriesDB = Connect("database.db", "user_id")
The dictionary should be inside an async function.字典应该在异步 function 中。
async def some_func():
countries_data = await countriesDB.to_dict("my_table_name", 123, "col1_name", "col2_name", ...)
You can insert any number of columns, or you can get all by specifying the column name as '*'您可以插入任意数量的列,也可以通过将列名称指定为“*”来获取所有列
countries_data = await countriesDB.to_dict("my_table_name", 123, "*")
so you now have made some changes to your dictionary and want to export it to sql format again?所以您现在已经对您的词典进行了一些更改并想再次将其导出为 sql 格式?
async def some_func():
...
await countriesDB.to_sql("my_table_name", 123, countries_data)
But what if you want a list of values for a specific column?但是,如果您想要特定列的值列表怎么办?
you can have a list of all values of a certain column.您可以列出特定列的所有值。
country_names = await countriesDB.select("my_table_name", "col1_name")
to limit your selection use limit
parameter.限制您的选择使用
limit
参数。
country_names = await countriesDB.select("my_table_name", "col1_name", limit=10)
you can also arrange your list
by using ascending
parameter and/or order_by
parameter and specifying a certain column to order your list accordingly.您还可以通过使用
ascending
参数和/或order_by
参数并指定特定列来相应地对列表进行排序来排列list
。
country_names = await countriesDB.select("my_table_name", "col1_name", order_by="col2_name", ascending=False)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.