您如何为具有多个 python 包的项目定义/组织 proto 文件？

Question

How do I define and organize gRPC proto files for a project with two (or more) python packages?

Suppose I have the following monorepo structure:

my_project/  
 |
 |-my_client_pkg/
 |  |_Dockerfile
 |  |_setup.py
 |  |_my_client/
 |     |___init__.py
 |     |_client.py
 |
  -my_server_pkg/
    |_Dockerfile
    |_setup.py
    |_my_server/
       |___init__.py
       |_server.py

I want to use gRPC to connect these two services together. I can add a protos/ directory, with the appropriate child directories like so:

my_project/  
 |
 |-protos/
 |  |_my_client_pkg/
 |  |  |_my_client.proto
 |  |
 |  |_my_server_pkg/
 |     |_my_server.proto
 |
 |-my_client_pkg/
 |  |_Dockerfile
 |  |_setup.py
 |  |_my_client/
 |     |___init__.py
 |     |_client.py
 |
  -my_server_pkg/
    |_Dockerfile
    |_setup.py
    |_my_server/
       |___init__.py
       |_server.py

and this works: I can generate the .py files correctly, with the correct import statements. The problem with this approach is that I must duplicate everything from my_client.proto into my_server.proto . If I need to make a change, it must be in both files, something is obviously error prone. It really seems to defeat the purpose of protobufs.

How should I be organizing my proto files? Is it possible to define the proto in one place, and then use the proto import statement to seamlessly duplicate the definitions? Should I be doing something completely different?

Edit: Here is an example proto file:

syntax = "proto3";

message AbcRequest {
  string utterance = 1;
}

message AbcResponse {
  string value = 2;
}

service Abc {
  rpc most_similar(AbcRequest) returns (AbcResponse) {}
}

I did try defining the proto in a single place, and then importing it:

my_project/  
 |
 |-protos/
 |  |-base.proto
 |  |_my_client_pkg/
 |  |  |_my_client.proto
 |  |
 |  |_my_server_pkg/
 |     |_my_server.proto

Where my_client.proto and my_server.proto both just have:

syntax = "proto3";
import "protos/base.proto";

However, when I generated the python code, the *_grpc.py file had nothing in it. Should I be doing something else for the importing to work like this?

Answer 1

protobuf import statements allow you to access definitions defined in other files, but will only generated code for messages/services defined in that file.
syntax = "proto3"; import "protos/base.proto";

Will generate empty files because there are no service/messages defined in this file.

The code for the file you imported will live in base_pb2_grpc .

I would recommend having a single my_service.proto file with all your definitions, and just referencing the generated _pb2_grpc.py file from both your my_client and my_server package.