如何调用具有 Python 和 TensorFlow 服务 Z20F35E630DAF44DB84ZC3 的特定 model 版本？

Question

I have a few machine learning models running via TensorFlow Serving on Kubernetes. I'd like to be able to have one deployment of a particular model, and then load multiple versions.

This seems like it would be easier than having to maintain a separate Kubernetes deployment for each version of each model that we have.

But it's not obvious how to pass the version or model flavor I want to call using the Python gRPC interface to TF Serving. How do I specify the version and pass it in?

Answer 1

For whatever reason, it's not possible to update the model spec in place as you're building the request to pull. Instead, you need to separately build an instance of ModelSpec that includes the version you want, and then pass that in to the constructor for the prediction request.

Also worth pointing out you need to use the Google-specific Int64Value for the version.

from google.protobuf.wrappers_pb2 import Int64Value
from tensorflow_serving.apis.model_pb2 import ModelSpec
from tensorflow_serving.apis import predict_pb2, get_model_metadata_pb2, \
                                    prediction_service_pb2_grpc
from tensorflow import make_tensor_proto
import grpc

model_name = 'mymodel'
input_name = 'model_input'
model_uri = 'mymodel.svc.cluster.local:8500'

X = # something that works

channel = grpc.insecure_channel(model_uri, options=MESSAGE_OPTIONS)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

version = Int64Value(value=1)
model_spec = ModelSpec(version=version, name=model_name, signature_name='serving_default')

request = predict_pb2.PredictRequest(model_spec=model_spec)
request.inputs[input_name].CopyFrom(make_tensor_proto(X.astype(np.float32), shape=X.shape))
result = stub.Predict(request, 1.0)
channel.close()