在一种模式下批处理Avro消息的好处？

Question

I am wondering how beneficial (performance and size-wise) it would be to batch Avro messages into one Avro message. It would have one schema for all the records instead of one per record. (Suppose schema management is not possible, so every time we send a message, we must send the schema along with it)

For example, say we have an Avro schema representing a 'person' that has 'height', 'weight' and 'age'. Suppose we have 10 people we want to record in Avro messages. We could either send 10 separate Avro messages, each with their schema in the metadata (taking up space), or 1 Avro message with an array of people and only one schema.

I am wondering how impactful this compression would be - what is the relative size of the schema and is it worth it to go to the trouble of architecting this compression? Or is it minimally effective, in which case it's easier to just send 10 separate messages?

Thanks in advance. Danielle

Answer 1

TL;DR: You very likely want to batch your messages, otherwise you would be better off emitting the data as JSON directly.

For example, let's use a Person record similar to what you suggest:

{
  "name": "Person",
  "type": "record",
  "fields": [
    {"name": "height", "type": "float"},
    {"name": "weight", "type": "float"},
    {"name": "age", "type": "int"}
  ]
}

Then, without compression:

• The schema itself is ~150 bytes.
• A random record (eg {"height": 213.47,"weight": 365.4,"age": 78} ) is:
  • ~10 bytes when binary-encoded.
  • ~50 bytes when JSON-encoded.

So, roughly, it's only worth using binary encoding (which requires including the schema) if you batch records 5+ at a time. Compression will also probably be in favor of JSON encoding, so you'll want to batch even more.

Of course, all this depends on your particular schema and values. For example if your values contain large arrays or strings, the relative cost of including the schema in each message will be smaller.