How to read and write many objects (or any data) to a buffer?

Question

I am writing a program to save some object (struct) to a buffer. I don't have experiment to write many objects to buffer and read these objects from the buffer. Any help would be appreciated. My code can write one item to object and I want write many objects to the buffer

struct PointFull {
    double lat;
    double lon;
};

PointFull item1;
PointFull item2;
PointFull item3;
PointFull item4;
PointFull item5;

void* buffer = malloc(sizeof (PointFull));
int fd = open("output", O_WRONLY | O_CREAT, S_IWUSR | S_IRUSR);
if (fd < 0) {
    printf("Error opening file\n");
    return 1;
}

//The below function can write only one item to buffer. How to write 5 item (from item1 to item5) to the buffer
//memcpy(buffer, &item1, sizeof (item));

write(fd, buffer, sizeof (item));

Now I have a file named "output" in hard disk and then I want read the file to test data.

    int fd2 = open("output", O_RDONLY, S_IWUSR | S_IRUSR);
if (fd2 < 0) {
    printf("Error opening file\n");
    return 1;
}
void* bufferRead;
bufferRead = malloc(5* sizeof (PointFull));
read(fd2, bufferRead,sizeof (PointFull));

At the moment, I have bufferRead contain 5 items but I dont know how to read buffer to insert data to struct??? Plz help me!

Answer 1

Well what you want to do is serialization. Say you have such structure:

struct PointFull {
    int lat;
    int lon;
};

and also

PointFull item1, item2;

The way you serialize it to buffer is:

unsigned char arr[20] = {0};
memcpy(arr, &item1.lat, sizeof(int));
memcpy(&arr[1 * sizeof(int)], &item1.lon, sizeof(int));
memcpy(&arr[2 * sizeof(int)], &item2.lat, sizeof(int));
memcpy(&arr[3 * sizeof(int)], &item2.lon, sizeof(int));

I am serializing like this because it is not good idea to directly write the structure like you suggest because of padding problems. The structures may have paddings and they may differ per system.

Now, you have the byte array (which contains two PointFull objects - for more objects you would follow similar approach) and you can use it in your write:

write(fd, arr, 20);

After reading the byte array you could reconstruct point objects by using similar memcpy calls like above (just destination would be the point object members now). But the problem with this is that integer serialization in binary is not portable (and floats moreover) - on different systems integers may have different size, different endianness. With floats additionally their representation may differ.

Anyway there is one way how to encode floats in binary here - check pack754 (and similar unpack) function. If you use that function for serializing floats in byte array and serialize each float separately like in this answer, then maybe you will be fine.

_{PS. here is post which explains serialization (for float encoding part you can use link in my answer).}

Answer 2

If you just want to write structures to a file, you can just write them directly :

write(fd, &item1, sizeof (item1));
write(fd, &item2, sizeof (item2));
...

Unless you really have a lot of those, it will perform alright, the OS itself will buffer the file system access.

If you really want to use a buffer, you can have a small class to write to the buffer :

class BufferWriter {
    char *ptr;
    int current;
public:
    BufferWriter( void *ptr ) : ptr((char*)ptr), current(0) {}
    template<typename T>
    void Write( const T &t ) {
        memcpy(&ptr[current], &t, sizeof(t) );
        current += sizeof(t);
    }
    int GetTotalLength() {
        return current;
    }
};

And then use it like that :

char *b = new char[ENOUGH];
BufferWriter buffer(b);

buffer.Write(item1);
buffer.Write(item2);
...

write(fd, b, buffer.GetTotalLength());
delete[] b;

You can add some code to check for buffer overflow if you want.

About the portability of the file you output (if you plan to transfer it to another device), you may want to use intX_t (ex: int32_t ) instead of int , short or whatever to be sure of the size. Also for ints, you may have to check the endianness of the system, but on personal devices it will always be little-endian. You don't have to worry about float and double , because all modern devices use the IEEE 754 norm, and even most of exotic devices also stick to this.