Thrift、Avro 和 ProtoBuf 数据治理

Question

We have a use case of data streaming from the main transactional system to other downstream such as data analytics and machine learning team.

One of the requirements are to ensure data governance that data source can control who can read which column, and potentially lifecycle of a data to ensure data siting in another domain gets purged should the source data removed it, such as if a user deletes the account, we need to make sure the data in all downstream gets removed.

While we are considering Thrift, Avro and ProtoBuf, what are the common frameworks that we can use for such data governance? Do any of these protocol supports metadata for such data governance around data authorization, lifecycle?

Answer 1

Let me get this straight:

protobuf is not a security device ; to someone with the right tools it is just as readable as xml or json, with the slight issue that it can be uncertain how to interpret some values;

It's not of a much difference than JSON nor XML. It is just an interface language. Sure, it has encoding , it is a bit different and a lot more customizable, but it does in no way confront security. It is up to you to secure the channel between sender and receiver.