简体   繁体   中英

Storing Keras model HDF5 file to SQL database

I have a Keras Neural Network model that gets updated regularly. For historical traceability, I would like to keep a copy of each trained model (architecture + weights + optimizer state) in a database. Keras will export an HDF5 file that includes all of the information about the model. Is there a way to convert this file into a format that could be stored in a SQL database record so that the full model could be retrieved/used later, even if that retrieval isn't super fast?

I am using Python3 to build/train the model and interact with the SQL database. Also, I'm using MS SQL Server for the database if that makes a difference.

Update:

Based on the comments, I am saving the HDF5 file and then reading that file back into Python like this:

filepath = 'C:\\path\\to\\file.h5'
model.save(filepath)
with open(filepath, "rb") as f:
    model_bytes = f.read()

When I try to INSERT the model_bytes value into a VARBINARY(MAX) column in MSSQL, I get the following error:

pypyodbc.DataError: ('22018', '[22018] [Microsoft][ODBC SQL Server Driver][SQL Server]Operand type clash: ntext is incompatible with varbinary(max)')

Any ideas on how to preprocess the byte data type in Python so that it will insert correctly in SQL?

When I print model_bytes to screen it looks like this:

b'\\x89HDF\\r\\n\\x1a\\n\\x00\\x00\\x00\\x00\\x00\\x08\\x08\\x00\\x04\\x00\\x10\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff@\\x13\\x01\\x00\\x00\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00`\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x88\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xa8\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x08\\x00\\x01\\x00\\x00\\x00\\x18\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x10\\x00\\x10\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x18\\x01\\x00\\x00\\x00\\x00\\x00\\x00TREE\\x00\\x00\\x01\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x18\\x00\\x00\\x00\\x00\\x00\\x00\\x18\\x00\\x00...

I was not able to find a clean way to do this. The way I solved it was to store the file name/path in the database along with some meta-data about the model. I stored the actual model (HDF5 file) in Google Cloud Storage. So when I want to retrieve the model, I query the DB to get the latest filename, then I download it from Google Cloud Storage.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM