How to write multiple messages in one protobuf file?

Question

Take the google's tutorial for examples:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

message AddressBook {
  repeated Person person = 1;
}   

It uses an AddressBook wrapper message as storing multiple Person messages. If I serialize or deserialize it I use methods like AddressBook.mergeFrom(FileInputStream) addressBook.build.().writeTo()

but it will obviously overflow if I have 20 million Person records. And also google says if I want to store multiple messages in one file without using the wrapper message I need to record each message's length, which is not possible for string types.

Is there a good way to serialize a large amount of messages in one file? And I am using Java by the way.

Answer 1

I haven't tried this, but I'd expect it to work:

Writing:

CodedOutputStream output = CodedOutputStream.newInstance(...);

while (...) {
    Person person = ...;
    output.writeMessageNoTag(person);
}

Reading:

CodedInputStream input = CodedInputStream.newInstance(...);
while (!input.isAtEnd()) {
    Person.Builder builder = Person.newBuilder();
    input.readMessage(builder, null); // Or specify extension registry
    Person person = builder.build();
    // Use person
}

Answer 2

In Java you can use writeDelimitedTo and parseDelimitedFrom methods for writing and reading multiple messages of the same type.

From MessageLite documentation :

Like writeTo(OutputStream) , but writes the size of the message as a varint before writing the data. This allows more data to be written to the stream after the message without the need to delimit the message data yourself. Use MessageLite.Builder.mergeDelimitedFrom(InputStream) (or the static method YourMessageType.parseDelimitedFrom(InputStream) ) to parse messages written by this method.

Example:

// writing
List<Person> addressBook = ...; // list of person to be stored
try(FileOutputStream output = new FileOutputStream(path)) {
  for (Person person: addressBook) {
    person.writeDelimitedTo(output); 
  }
}

// reading
try (FileInputStream input = new FileInputStream(path)) {
  while (true) {
    Person person = Person.parseDelimitedFrom(input);
    if (person == null) { // parseDelimitedFrom returns null on EOF
      break;
    }

    // use person
  }
}

Answer 3

The key is to open the file in append mode.

FileOutputStream output = new FileOutputStream(file, true);

So, the following will be the complete solution.

// writing
List<Person> addressBook = ...; // list of person to be stored
try(FileOutputStream output = new FileOutputStream(path, true)) {
  for (Person person: addressBook) {
    person.writeDelimitedTo(output); 
  }
}

// reading
try (FileInputStream input = new FileInputStream(path)) {
  while (true) {
    Person person = Person.parseDelimitedFrom(input);
    if (person == null) { // parseDelimitedFrom returns null on EOF
      break;
    }

    // use person
  }
}

Hope this saves somebody a few hours of time.