酸洗，压缩Python对象并存储在SQL中时信息丢失

Question

I am using the pickle and zlib libraries to serialize and compress 3 objects for storage in TSQL as nvarchar(MAX), using pyodbc. When retrieving these objects from SQL on the other end, 2 successfully translate back to their original form (2d numpy array and sklearn.StandardScaler). The third object (sklearn.GradientBoostingRegressor) does not translate back. Below is a summary of the code used. Should also note that I'm using Pandas for some SQL work.

Below is the pickling, compression and upload to SQL code:

model_pickle = zlib.compress(pickle.dumps(est))
scaler_pickle = zlib.compress(pickle.dumps(scaler),1)
boundary_pickle = zlib.compress(pickle.dumps(bounds),1)

cnxn = pyodbc.connect('driver={SQL Server};server=XXX;database=XYZ;trusted_connection=true')
cursor = cnxn.cursor()

cursor.execute("""INSERT INTO AD_RLGNMNT_MDL_PICKLE (CHAIN_NUM,AD_LENGTH,GL_CODE,AD_TYPE,MODEL_PICKLE,SCALER_PICKLE,WFA_TEST,BOUNDS_PICKLE) VALUES (?,?,?,?,?,?,?,?);""",
               (chain_num.astype(np.int32),ad_length.astype(np.int32),gl_code.astype(np.int32),ad_type,model_pickle,scaler_pickle,WFA,boundary_pickle))
cursor.commit()
cursor.close()
cnxn.close()

Below is the code for pulling from SQL, decompressing and unpickling the objects:

model_pickle = pd.read_sql("""SELECT A.MODEL_PICKLE from AD_RLGNMNT_MDL_PICKLE A WHERE A.CHAIN_NUM = %s and A.ad_length = %s and A.GL_CODE = %s and A.AD_TYPE in ('%s');"""
            % (Current_chain_num, Current_ad_length, Current_GL, Current_ad_type),cnxn)
scaler_pickle = pd.read_sql("""SELECT A.SCALER_PICKLE from AD_RLGNMNT_MDL_PICKLE A WHERE A.CHAIN_NUM = %s and A.ad_length = %s and A.GL_CODE = %s and A.AD_TYPE in ('%s');"""
            % (Current_chain_num, Current_ad_length, Current_GL, Current_ad_type),cnxn)
bounds_pickle = pd.read_sql("""SELECT A.BOUNDS_PICKLE from AD_RLGNMNT_MDL_PICKLE A WHERE A.CHAIN_NUM = %s and A.ad_length = %s and A.GL_CODE = %s and A.AD_TYPE in ('%s');"""
            % (Current_chain_num, Current_ad_length, Current_GL, Current_ad_type),cnxn)

combos_list.set_value(i, 'model', model_pickle[['MODEL_PICKLE']].iloc[0].values[0])
combos_list.set_value(i, 'scaler', scaler_pickle[['SCALER_PICKLE']].iloc[0].values[0])
combos_list.set_value(i, 'bounds', bounds_pickle[['BOUNDS_PICKLE']].iloc[0].values[0])

model  = pickle.loads(zlib.decompress(model_pickle[['MODEL_PICKLE']].iloc[0].values[0]))
scaler = pickle.loads(zlib.decompress(scaler_pickle[['SCALER_PICKLE']].iloc[0].values[0]))
bounds = pickle.loads(zlib.decompress(bounds_pickle[['BOUNDS_PICKLE']].iloc[0].values[0]))

When I run the "model = pickle.loads" line, I receive the following error (again, the last two lines work correctly):

error: Error -5 while decompressing data: incomplete or truncated stream

I have scoured SO and the web for solutions, and have tried countless variations of the above code. I have compared the raw ASCI input and output, and input is 203,663 characters long, while the output is 203,649. The output is missing an "\\xb3" at the end, as well as an "xaa\\\\" and an "\\\\n" in the middle.

Is there something unique about the GradientBoostingRegressor object? Pulling my hair out here.

Answer 1

You need use binary BLOB to store your data in database (more complexe to implement)

or

Simply use Character CLOB to store your data after having made a base64 encoding

import base64

model_pickle=base64.b64encode(zlib.compress(pickle.dumps(est)))

model=pickle.loads(zlib.decompress(base64.b64decode(model_pickle[['MODEL_PICKLE']].iloc[0].values[0])))

note : base64 increase data volume (+33%), but I think it's not very significant

Answer 2

**Update: This only worked for one example. The selected answer above by user indent fixed my problem.

Figured it out! Modified the first line of code to the following, which includes selecting a more recent pickle protocol.

model_pickle = zlib.compress(pickle.dumps(est,protocol = 2))