棉花糖：将字符串数据库列映射到字典字段

Question

I have the following database table in SQL server:

COLUMN_NAME              ORDINAL_POSITION DATA_TYPE       
----------------------------------------------------
id                       1 varchar
collection_id            2 varchar
created_at               3 datetimeoffset  
mimetype                 4 varchar
size                     5 int
file_hash                6 varchar
storageKey               7 varchar
extra_data               8 varchar

this is mapped to a Marshmallow Schema in flask app like so:

class ImageDisplaySchema(Schema):
    id = fields.Str()
    created_at = fields.Str()
    collection_id = fields.Str()
    mimetype = fields.Str()
    size = fields.Int()
    extra_data = fields.Dict()

I'm struggling to find the right pre_dump / pre_load helpers to serialize/deserialize the extra_data column which is saved in the database as a json string. I've tried a few variations but always end up with a serialization exception. Here's my current version of the pre_dump helper:

    @pre_dump
    def serialize_extra_data(self, data, many):
        """This will alter the data passed to ``dump()`` before Marshmallow
        attempts serialization.
        """
        print(type(data), type(data.extra_data), data.extra_data)
        extra_data = data.extra_data
        data.extra_data = json.loads(extra_data)
        return data

But this attempts to push a dictionary rather than a string into the DB:

cartridgeocr-annotations-1           |   File "src/pymssql/_mssql.pyx", line 1976, in pymssql._mssql._quote_or_flatten
cartridgeocr-annotations-1           | ValueError: expected a simple type, a tuple or a list

On the other hand if I change the loads to a dumps , some other serialization point complains that I don't have a dictionary in the field:

cartridgeocr-annotations-1           | ValueError: dictionary update sequence element #0 has length 1; 2 is required

The latter exception occurs when the flask app attempts to dump the object in a query response:

 return schemas.ImageDisplaySchema().dump(image_in_db), 201

I'm looking for a working example of how to seamlessly convert the db string to a dictionary and back via Marshmallow

Answer 1

When using SQLAlchemy, I usually return a model object as an output of the load of the marshmallow schema by using post_load . So the code could like this:

import json
from datetime import datetime
from dataclasses import dataclass

from marshmallow import Schema, fields, pre_dump, post_load


@dataclass
class DbModel:
    id: str
    collection_id: str
    created_at: datetime
    mimetype: str
    size: int
    file_hash: str
    storage_key: str
    extra_data: str


class ImageDisplaySchema(Schema):
    id = fields.Str()
    created_at = fields.Str()
    collection_id = fields.Str()
    mimetype = fields.Str()
    size = fields.Int()
    extra_data = fields.Dict()

    @pre_dump
    def serialize_extra_data(self, data, many):
        """This will alter the data passed to ``dump()`` before Marshmallow
        attempts serialization.
        """
        print(type(data), type(data.extra_data), data.extra_data)
        extra_data = data.extra_data
        data.extra_data = json.loads(extra_data)
        return data
    
    @post_load
    def deserialize_to_model(self, data, **kwargs):
        # dumps : dict => str
        extra_data = json.dumps(data.pop("extra_data"))
        return DbModel(
            **data,
            extra_data=extra_data,
            file_hash=None,
            storage_key=None
        )


db = DbModel("id", "collection_id", datetime.now(), "mimetype", 1, "hash", "storage", '{"extra": "data", "key": 1}')

# dump : object => dict
dumped = ImageDisplaySchema().dump(db)
assert isinstance(dumped["extra_data"], dict)

# load : dict => object
to_db = ImageDisplaySchema().load(dumped)
assert isinstance(to_db.extra_data, str)

Side note:

On the other hand if I change the loads to a dumps, some other serialization point complains that I don't have a dictionary in the field

As per the documentation of json stdlib:
dumps
Serialize obj to a JSON formatted str using this conversion table. The arguments have the same meaning as in dump().

loads
Deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object using this conversion table.

So doing a json.dumps on a str doesn't work as the function expects a dict .