What's the fastest way to serialize and deserialize a list of objects in Python 2.7?

Question

I need the fastest method to deserialize a large list (20000 items) of a custom class where the class only contains integers, floats and string attributes.

Should I use pickle, cPickle, json or something else?

So far I tried pickle which crashes when I try to load. I tried to use json.dumps and it errors out saying my custom type is not serializable.

class MyType
...

list = []
for i in range(10000):
    list.append(MyType(i))

list.serialize?

Answer 1

Given that the class only contains simple types like integers, floats and strings, you could use the struct module and it should be fast. Something like this for two integers (32 bits each), a float (double), and a string (up to 20 characters):

codec = struct.Struct('<iid20s')
size = codec.size
buf = bytearray(size * len(items))

offset = 0
for item in items):
    codec.pack_into(buf, offset, item.int1, item.int2, item.float1, item.string1)
    offset += size

Now buf contains all the data and can be written to a file.