Google Protobuf基于C ++的python实现的性能

Question

I am using Google Protobuf in my Python application. Experimenting with the protobufs I found that Protobuf Message Creation is much slower in CPP based python implementation as compared to Python based Python implementation.

Message creation with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp was upto 2-3 times slower as compared to pure python based Protobuf Message Creation.

Is this expected? I found that SerializeToString and ParseFromString both are faster in the cpp verison. The difference in each case widens as the size of the Message increases.

I am using python's standard timeit module to time these tasks.

(Using Version 2.4.1 of google protobuf)

Answer 1

Yes, I believe this is expected. The pure-Python implementation stores all the fields in a dict. To construct a new message, it essentially just creates an empty dict, which is very fast. The C++ implementation actually initializes a C++ DynamicMessage object under the hood and then wraps it. DynamicMessage actually initializes all of the fields upfront, so even though it's implemented in C++, it's "slower" -- but this upfront initialization makes later operations faster.

I believe you can improve performance further by compiling C++ versions of your protobuf objects and loading them in as another extension. If I recall correctly, the C++-backed Python protobuf implementation will then automatically used the compiled versions rather than DynamicMessage .