将元组列表（int、float）写入流而不转换为字符串

Question

I have a list in Python that consists of tuples that have the following format: (int, float). I want to write this list to a io byte or io raw stream without having to convert the ints and/or floats to a string. How can I do this? Thanks.

Answer 1

There are many formats which can be used to serialize Python objects into bytes. There are pros and cons for each of them.

If the data has only a list of tuples of integers and flaots, that make the job rather simple.

Let's assume, this is the data:

data = 100 * [(1, 1.111), (18, 1.234), (555555, 0.001), (-1, 1e70)]

Which of them falls into the category of "strings" is not clear to me. The most obvious "string" format would be str(data) . How big is it?

>>> len(str(data))
5500

This takes up 5500 bytes. The question asks for something more compressed. So, we're looking for something much shorter than 5500 bytes.

JSON is a very popular format (it is also a string). How big is it?

>>> len(json.dumps(data))
5500

This has the same size (5500 bytes), but at least it is well defined. Can it be smaller? How about a BZipped JSON ?

>>> len(bz2.compress(json.dumps(data).encode('utf-8')))
131

That is much better!

This was probably very good because of a repeating pattern. Is there a format which does not use zipping? Maybe pickle ?

>>> len(pickle.dumps(data))
862

Not as good as zip (of course!), but still good.

Could we make a BZipped pickle ?

>>> len(bz2.compress(pickle.dumps(data)))
155

Better, but there is no reason for it to be better than BZipped JSON.

How about some other format? You could convert each tuple to the equivalent of this C structure , using the struct module:

struct {
    int i;
    double f;
};

However, then you'd have to know how big the int can be. Python int can be as big aas you want, but if you eg know that all numbers are between 0 and 255, you just need one byte. For the float, you need 64 bits (ie 8 bytes), or you lose precision. So, this will go up to about 1000 bytes. Not very good.

There are also other built-in options documented in Python's documentation on Persistence .

You can also invent your own format.

In the end, you have to decide what suits you best.

Answer 2

You can dump integers and floats into bytes directly really easily using the struct module .

>>> import struct
>>> data = [(2, 1.0), (3, 2.0), (25, 55.5)]
>>> for tup in data:
    bytes_data = struct.pack("<ld", *tup)
    print(bytes_data)


b'\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0?'
b'\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@'
b'\x19\x00\x00\x00\x00\x00\x00\x00\x00\xc0K@'

As an aside the string I use as the first argument to the pack function is a format identifier that tells you what type and size of each number, in this case l is a long signed int, d is a float double.