I have a Keras Neural Network model that gets updated regularly. For historical traceability, I would like to keep a copy of each trained model (architecture + weights + optimizer state) in a database. Keras will export an HDF5 file that includes all of the information about the model. Is there a way to convert this file into a format that could be stored in a SQL database record so that the full model could be retrieved/used later, even if that retrieval isn't super fast?
I am using Python3 to build/train the model and interact with the SQL database. Also, I'm using MS SQL Server for the database if that makes a difference.
Based on the comments, I am saving the HDF5 file and then reading that file back into Python like this:
filepath = 'C:\\path\\to\\file.h5'
model.save(filepath)
with open(filepath, "rb") as f:
model_bytes = f.read()
When I try to INSERT
the model_bytes
value into a VARBINARY(MAX)
column in MSSQL, I get the following error:
pypyodbc.DataError: ('22018', '[22018] [Microsoft][ODBC SQL Server Driver][SQL Server]Operand type clash: ntext is incompatible with varbinary(max)')
Any ideas on how to preprocess the byte data type in Python so that it will insert correctly in SQL?
When I print model_bytes
to screen it looks like this:
b'\\x89HDF\\r\\n\\x1a\\n\\x00\\x00\\x00\\x00\\x00\\x08\\x08\\x00\\x04\\x00\\x10\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff@\\x13\\x01\\x00\\x00\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00`\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x88\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xa8\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x08\\x00\\x01\\x00\\x00\\x00\\x18\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x10\\x00\\x10\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x18\\x01\\x00\\x00\\x00\\x00\\x00\\x00TREE\\x00\\x00\\x01\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x18\\x00\\x00\\x00\\x00\\x00\\x00\\x18\\x00\\x00...
I was not able to find a clean way to do this. The way I solved it was to store the file name/path in the database along with some meta-data about the model. I stored the actual model (HDF5 file) in Google Cloud Storage. So when I want to retrieve the model, I query the DB to get the latest filename, then I download it from Google Cloud Storage.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.