zlib decompress header check error

Question

I have an issue with a dictionary I'm compressing and storing to a postgres database. I can compress the dictionary and decompress it without fault but when I insert it into the database, and then select it back and try to decompress the data I get an error:

zlib.error: Error -3 while decompressing data: incorrect header check

Here's a test script I wrote to emulate all the parts:

import zlib
import cPickle
import psycopg2

try:
    db = psycopg2.connect( database='*****', user='*****', password='*****',host='******')
    cursor = db.cursor()

except:
    print 'no db'

atom_id = 166503
params = {'submission': 'RVBV4SXLVVDNAAKIG2LIJKZQ', 'campaign': 'p3percybot', 'dob': '2011-03-11', 'rndrpath': '/mnt/webservices/store/p3percybot/rndr/2011-03-11/RVBV4SXLVVDNAAKIG2LIJKZQ', 'outpath': '/mnt/webservices/store/p3percybot/out/2011-03-11/RVBV4SXLVVDNAAKIG2LIJKZQ', 'srcpath': '/mnt/webservices/store/p3percybot/src', 'root': '/mnt/webservices/store', 'inpath': '/mnt/webservices/store/p3percybot/in/2011-03-11/RVBV4SXLVVDNAAKIG2LIJKZQ'}

print params
params_list = []
for k in params :
        param_name = k
        param_value = cPickle.dumps(params[k])
        param_value = zlib.compress(param_value,9)
        param_value = buffer(param_value)
        params_list.append((param_value,atom_id, param_name))

print params_list

sql = 'UPDATE atomparams set value = %s where atomid=%s and name=%s'
cursor.executemany(sql, (params_list))

sql = 'SELECT name, value FROM atomparams WHERE atomid=%s'
cursor.execute(sql, (atom_id,))
result = cursor.fetchall()

print '\n-----------------result-----------------'
print result
for data in result:
   print data[0]
   data_string = zlib.decompress(data[1])
   print data_string

Open to any suggestions as to why this is getting messed up in the db. I should note the field type storing the value is of type bytea

Thanks in advance!

Answer 1

The binary string requires escaping

psycopg2

http://initd.org/psycopg/docs/usage.html#adaptation-of-python-values-to-sql-types

http://initd.org/psycopg/docs/module.html#psycopg2.Binary

Postgres Binary String

https://www.postgresql.org/docs/current/static/datatype-binary.html#DATATYPE-BINARY-SQLESC