Serving a Keras model with Tensorflow Serving

Question

Tensorflow 1.12 release notes states that: "Keras models can now be directly exported to the SavedModel format(tf.contrib.saved_model.save_keras_model()) and used with Tensorflow Serving" . So I gave it a shot -

I have exported a simple model with this op using a single line. However, Tensorflow serving doesn't recognize the model. I guess the problem is with the docker call, and maybe with a missing 'signature_defs' in the model definition. I would be thankful for info regarding the missing steps.

1. Training and exporting the model to TF serving :

Here is the code based on Jason Brownlee's first NN (chosen thanks to its simplicity)

(the training data, as a short CSV file, is here ):

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.contrib.saved_model import save_keras_model
import numpy

# fix random seed for reproducibility
numpy.random.seed(7)

# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]

# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(X, Y, epochs=150, batch_size=10)

# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

# calculate predictions
predictions = model.predict(X)
# round predictions
rounded = [round(x[0]) for x in predictions]
print(rounded)

# Save the model for serving
path = '/TensorFlow_Models/Keras_serving/saved_model' # full path of where to save the model
save_keras_model(model, path)

2. Setting up Tensorflow Server :

The server can be set via docker or by its own build. TF recommends docker ( TF ref ). Following this, and based on TF blog and TF Serving Tutorial :

1. Install Docker (from here )
2. Get the latest TF serving version:

docker pull tensorflow/serving

3. Activate TF serving with this model ( TF ref ):

docker run -p 8501:8501 --name NNN --mount type=bind,source=SSS,target=TTT -e MODEL_NAME=MMM -t tensorflow/serving &

I would be happy if one could confirm:

• NNN - the docker container name - which is used, for instance, to kill the process. It can be set arbitrarily (eg to: mydocker).
• MMM - the name of the model, which seem to be set arbitrarily.
• SSS - the folder where the model is located, full path.
• TTT - What should be this set to ?

3. the client

The server can get requests either over gRPC or RESTful API. Assuming we go with RESTful API, the model can be accessed by using curl ( here is a TF example ). But how do we set the input/output of the model? does SignatureDefs needed ( ref )?

All in all , while "Keras models can now be directly exported to the SavedModel format(tf.contrib.saved_model.save_keras_model()) and used with Tensorflow Serving" , as stated in TF1.12 release notes, there is a way to go in order to actually serve the model. I would be happy for ideas on completing this.

Answer 1

You are all correct about NNN and SSS. NNN can be arbitrary, if not specified, docker will give it a random name.

For MMM, better give it a meaningful name.

For TTT this is general about docker run command, and you can refer docker doc . This is where you map(bind) SSS inside the container, usually set to /models/$MODEL_NAME . If you get into this container and open /models/$MODEL_NAME , you will see the version folder(s) just as in SSS.

The input of RESTful API is the same as the input to the model in TensorFlow code, in your example is X = dataset[:,0:8] .

If you didn't define signature when saving the model like the example in doc , then it's not necessary in serving.

Answer 2

If you want to create an API endpoint server with Keras out of the box. I recommend you to check out BentoML( https://github.com/bentoml/bentoml ).

It is an open source python library to make it easy to serve and deploy machine learning models in the cloud. disclosure: I am one of the creator of BentoML .

Here is an exammple of Keras model using BentoML ( https://github.com/bentoml/gallery/blob/master/keras/fashion-mnist/keras-fashion-mnist.ipynb )

With few line of codes, you could package your model, preprocessing code and dependencies into a standardized format that could distribute to different formats (Docker, CLI tool, Spark UDF, and etc) and deploy them to different cloud platforms(Sagemaker, Lambda, Kubernetes)

After you finished training your model in Jupyter notebook. You can add BentoService spec to create your model in standardized bundle format

%%writefile my_model.py

from bentoml import api, artifacts, env, BentoService
from bentoml.artifact import KerasModelArtifact
from bentoml.handlers import DataframeHandler

@env(pip_dependencies=['keras', 'tensorflow==1.14.0'])
@artifacts([KerasModelArtifact('model')])
class KerasModelService(BentoService):

    @api(DataframHandler)
    def predict(self, df):
        return self.artifacts.model.predict(df)

In the next cell you could pack your model and save it.

# 1) import the custom BentoService defined above
from my_model import KerasModelService

# 2) `pack` it with required artifacts
keras_svc = KerasModelService.pack(model=model)

# 3) save your BentoSerivce to file archive
saved_path = keras_svc.save()

These will packaged your model, preprocessing code, dependencies and also generate Dockerfile and other for you.

With this bundle, you could easily start a docker container with REST API endpoints.

cd saved_path && docker build -t keras-model .

Feel free to ping me, if you have any questions.

Answer 3

Thanks for your question, more or less is linked with mine tensorflow-serving signature for an XOR

I add exactly your doubts about the TTT