Compress text to store in mysql database, does it have to be base64

Question

I got a code from here that defines a compressed text field. I need to do this because I'm storing too much text and my database is too big. The problem is that the code doesn't have any documentation and it's confusing.

Particularly, I have modified the code a little, in here:

def get_prep_value(self,value):
    if not value:
        return value
    try:
        tmp = value.encode('utf-8').encode('bz2')
    except Exception:
        return value
    else:
        if len(tmp) > len(value):
            return value
        return tmp

In the original code, they encode to base64 after bz2, which it shows not to optimize but I was wondering if there might be another reason to do that? btw. I'm using MySql back-end

I also removed lines 11-15 that didn't make sense to me.. why would you decode in here?

Answer 1

Base64-encoding the data guarantees that the resulting data will be safe to insert into a text-only column (while sacrificing some of the compression that bzip2 offers). The author must have had a requirement to insert the data into a text column. If you're using a BLOB type of column, you don't need to worry about the base64 part (and you'll get more compression).

The linked example seems a bit roundabout in light of the fact that MySQL supports gzip compression natively. See the MySQL documentation regarding compression and encryption functions , particularly COMPRESS() and UNCOMPRESS() . If you have BLOB columns which can store binary data, these will happily store your compressed data.

The downside to this approach is that the uncompressed data needs to make a trip to the server where it is compressed (or uncompressed before shipping over the network back to the client). This might have provided the motivation behind the author's original snippet.