Python extension module using TensorFlow C API

Question

I've written a Python extension module that uses TensorFlow through the C API. I installed the API as described in https://www.tensorflow.org/install/lang_c . On its own, my module works correctly. But if I import my extension module and also import tensorflow, Python crashes with this error.

[libprotobuf ERROR external/protobuf_archive/src/google/protobuf/descriptor_database.cc:58] File already exists in database: tensorflow/core/protobuf/master.proto
[libprotobuf FATAL external/protobuf_archive/src/google/protobuf/descriptor.cc:1370] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
Abort trap: 6

I believe this happens because the C API includes its own complete copy of TensorFlow, so now I get two different copies loaded into the same process at the same time.

What is the solution to this? How can I have Python code that uses TensorFlow, and also invokes C code that uses TensorFlow?

Answer 1

TensorFlow uses Protobuf and it stores data globally, by registering message types with specific names.

In your case, both TensorFlow and your extension are linked with Protobuf (probably statically) and upon initialization try to register all the necessary messages. The first (TensorFlow) succeeds, and the second fails with this very message, that specifies that someone (TensorFlow..) has already registered such a message.

Try not to link with the TensorFlow library (which is dependent on Protobuf) and load it dynamically.

This may be done using dlopen() but then you'll have to manually get all TF methods using dlsym() which is a bit different than what you'd like.

If you compile a Python extension, I guess you're compiling a shared-object so I'd expect it to be dynamically linked with TensorFlow. What is the command line you used for compilation? Did you use -fPIC ?