How to reduce size of my custom python type?

Question

I created my own custom Python type. It stores an internal array of bytes. Even when there's no bytes in the array, the object takes up 32 bytes. How can I reduce that?

Here's how I defined my type:

typedef struct {
    PyObject_VAR_HEAD
    uint8_t *data;
} BitStream;

static PyTypeObject BitStreamType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "my_module.BitStream",
    .tp_dealloc = (destructor) _BitStream_dealloc,
    .tp_basicsize = sizeof(BitStream),
    .tp_itemsize = sizeof(uint8_t),
    .tp_flags = Py_TPFLAGS_DEFAULT,
};

I exposed a method to create an instance of it with a certain number of bytes in the data array:

static PyObject *
of_bytes(PyObject *self, PyObject *obj)
{
    unsigned long size = PyLong_AsUnsignedLong(obj);
    BitStream *bstream = (BitStream *) BitStreamType.tp_alloc(&BitStreamType, size);
    if (bstream == NULL) {
        return NULL;
    }

    return (PyObject *) bstream;
}

I would love for of_bytes(0) to have a size as close to 0 as possible, and of_bytes(1) to have a size close to 1, etc.

Unfortunately:

>>> sys.getsizeof(0)
24
>>> sys.getsizeof(of_bytes(0))
32

Why is it larger than int ?

How can I reduce its size?

It only becomes consistently smaller than int when both objects have to represent numbers with more than 937 bits:

>>> sys.getsizeof(2 ** 937)
152
>>> sys.getsizeof(of_bytes(math.ceil(937 / 8)))
150

For any more bytes/bits than this, int never becomes smaller than it again.

But I want it to be smaller than, or at least the same size as, int for any number of bits stored.

Alternatively, since all I need is the byte array uint8_t *data , is there another way I can return such an array to my code without a 32-byte PyObject wrapper?

Answer 1

Now I achieve sizes of 24 bytes + however many bytes are in data .

Have to use uint8_t data[1] intstead of uint8_t *data .

Then need to allocate memory with PyObject_InitVar .

typedef struct {
    PyObject_VAR_HEAD
    uint8_t data[1];
} BitStream;

static PyTypeObject BitStreamType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "my_module.BitStream",
    .tp_dealloc = 0,
    .tp_free = PyObject_Del,
    .tp_basicsize = offsetof(BitStream, data),
    .tp_itemsize = sizeof(uint8_t),
    .tp_flags = Py_TPFLAGS_DEFAULT,
};

static BitStream *
_BitStream_New(size_t size)
{
    BitStream *bstream = (BitStream *) PyObject_Malloc(offsetof(BitStream, data) + size);
    if (!bstream) {
        PyErr_NoMemory();
        return NULL;
    }
    PyObject_InitVar((PyVarObject*)bstream, &BitStreamType, size);
    return bstream;
}

It's been working so far.