C++ protobuf: how to write multiple messages into file by “SerializeToOstream()”

Question

I found when I use SerializeToOstream(), only the last message was written to file, and previous ones are all overwritten by later calls. Like this:

I've my hello.proto:

message hello
{
    required int32 f1=1;
    required int32 f2=2;
    optional int32 f3=3;
}

Then I had it compiled and used in one cpp file:

#include "hello.pb.h"
#include<fstream>
#include<iostream>
using namespace std;
int main()
{
    fstream fo("./hello.data",ios::binary|ios::out);
    hello p1,p2,p3;
    p1.set_f1(1);
    p1.set_f2(2);
    p2.set_f1(3);
    p2.set_f2(4);
    p3.set_f1(5);
    p3.set_f2(6);
    p1.SerializeToOstream(&fo);
    p2.SerializeToOstream(&fo);
    p3.SerializeToOstream(&fo);
    fo.close();

    fstream fi("./hello.data",ios::binary|ios::in);
    hello pi;
    pi.ParseFromIstream(&fi);
    cout<<pi.f1()<<pi.f2()<<endl;
    return 0;
}

Well I found only "p3" was writen into this "hello.data", the p1 and p2 were overwritten in some way?

Why protobuf only write the last message? We usually use protobuf to communicate multiple messages, right? So how to write multiple messages into one file? How to correct my program?

Answer 1

The problem here is that protocol buffers are not self-delimiting; to parse a message you must know in advance how many bytes to read. Because of the way the wire format works, you can concatenate two (or more) serialized protocol buffers and still have a valid message; for any given singular field, later values take precedence over earlier ones. So what is happening is that ParseFromIstream is reading all three serialized messages as if they were one message, and the last message is effectively taking precedence over the two previous ones.

To fix this, you have to include some kind of message-length prefix for each message. Typically this is done by prefixing each message with a varint storing its length. This code from Kenton Varda is a good example of exactly how to do this in C++.