how to save a huge amount of integers in a data file without consuming too much disk space in c?

Question

I'm running a simulation with 20000 sweeps, and for every sweep, I need to save a very large array of integers with the values 1, and -1 to a data file. The number of integers printed can reach 1e11.

my problem is that the size of the data file becomes huge (up to 50GB)

I'm currently using fprintf with int datatype.

does anyone know a way to save these values without taking that much disk space?

Thanks in advance

Answer 1

1. If your numbers are only -1 and 1 then it can be represented in one bit. If number 0 also exists you will need an additional one bit to represent 3 values (2 bits can actually hold 4 distinct values).
2. 1e11 bits can be represented in 12.5e9 bytes. If two bits are required then 25.0e9 bytes will be needed. It is not a problem for most modern computers. You may even have enough RAM to keep it in RAM or map it.

for single bit:

int getVal1bit(const void *buff, size_t index)
{
    const unsigned char *ucbuff = buff;

    ucbuff += index / CHAR_BIT;

    return *ucbuff & (1 << (index % 8)) ? 1 : -1;
}

void putVal1bit(void *buff, size_t index, int val)
{
    unsigned char *ucbuff = buff;

    ucbuff += index / CHAR_BIT;

    if(val < 0) *ucbuff |= (1 << (index % 8));
    else *ucbuff &= ~(1 << (index % 8));
}