Python Pickle需要多少内存？

Question

I have some pandas DataFrames which I can save to disk with .to_pickle() . Such an object is 200k-700k.

I see from memcache.py in the Python-memcache github project , it pickles objects and compresses them before caching.

By default, memcached only allows values up to 1MB. I find that trying to cache my 200k DataFrames works fine, but the 600k ones don't get set at the Python memcache level (the client doesn't even issue the set command unless I use -I on memcached and set memcache.SERVER_MAX_VALUE_LENGTH accordingly for my Python client).

Storing ~100 such dataframes to memcache with -I 5m allows them all to fit and takes up 36MB (36212 bytes) on disk with written pickle files. Per memcached stats command, I see nearly 3x the bytes were written,

STAT bytes_read 89917017
STAT bytes_written 89917211
...
STAT bytes 53022739

It is then strange that only 53MB are being stored if 89MB were written.

If I alter my memcaching code to pickle the DataFrames first (ie write to a tempfile with .to_pickle() , read that tempfile to store to memcache), I see the data sizes per memcache stats matching what's on disk when i store the same files.

STAT bytes_read 36892901
STAT bytes_written 36893095
...
STAT bytes 36896667

What is the ratio of memory used to store a pickled object compared to its size on disk? And why wouldn't python memcache do a similarly efficient job of converting DataFrames to smaller pickle sizes as using .to_pickle() ?

Answer 1

It seems that python-memcache uses ASCII encoding when it pickles objects, while pandas' to_pickle() uses pickle v2 in binary encoding which is smaller. Were I to export my dataframes to CSV per @BrenBarn's suggestion, I'd get files slightly larger than the binary dataframes, but still ~1/3 the size of an ASCII-pickled dataframe.

My workaround to use Pandas to do the binary pickling before memcaching goes like this (I also added namespace arg much like Google App Engine to help ensure key uniqueness when using the same memcache for different applications):

import memcache
import tempfile
import pandas as pd
mc_client = memcache.Client(['localhost:11211'], debug=0)
def mc_get(key, namespace):
    """ Gets pickle from Memecache and converts to a dataframe
    """
    data = mc_client.get('{}_{}'.format(namespace, key))
    if data is None:
        return
    temp_file = tempfile.NamedTemporaryFile()
    temp_file.write(data)
    temp_file.flush()
    return pd.read_pickle(temp_file.name)

def mc_set(key, df, namespace):
    """ Convert dataframe to dict and store it to memcache
    """
    temp_file = tempfile.NamedTemporaryFile()
    df.to_pickle(temp_file.name)
    temp_file.flush()
    data = temp_file.read()
    mc_client.set('{}_{}'.format(namespace, key), data)

It may seem like this usage of a tempfile would slow things down as it writes to disk, but test show it being 2x as fast as just storing pickles to disk and loading them from there.

Looking at the python-memcached code, I see that one can use set(min_compress_len=X) to trigger python-memcache to compress values before setting them. Using this method reduced memory to 40% of what my pre-pickling trick did.

Lastly, the python-memcached constructor takes a pickleProtocol arg which, if set to 2 , will use the same pickling protocol Pandas' to_pickle() does.

Combining pickleProtocol=2 with min_compress_len=1 (to cause compression always) caused memory usage to be about 25% of what it was with binary pickling alone, and the overhead to compress added about 13% to the run time to write all my dataframes to memcache.