如何序列化 Tensorflow 服务请求以减少推理/预测延迟？

Question

I have exported a TF SavedModel from an Estimator using TensorServingInputReceiver as follows:

def serving_input_fn():
    input_ph = tf.placeholder(tf.float32, shape=[None, 3, 224, 224], name = 'image_batches')
    input_tensors = input_ph
    return tf.estimator.export.TensorServingInputReceiver(input_tensors, input_ph)

and export the SavedModel as follows:

warm_start = tf.estimator.WarmStartSettings(CKPT_DIR)
    classifier = tf.estimator.Estimator(model_fn = model_fn, warm_start_from = warm_start)
    classifier.export_savedmodel(export_dir_base = SAVED_MODEL_DIR, serving_input_receiver_fn = serving_input_fn)

However, when I use this SavedModel to perform predictions in Tensorflow Serving:

json_dict = {'signature_name': 'serving_default', 'instances': data}

where data is a numpy array, I obtain speed of only about 1/5 to 1/6 of local direct inference using the SavedModel.

Currently I'm thinking the problem may be the serialization part of the request in JSON as suggested here . So does anyone know how to perform serialization of the request before sending, or have any suggestions as to why the inference speed using TF Serving is much slower than direct inference?

Answer 1

I have been facing the same situation, jason.dumps causes 90% of delay, if the image size is reduced to say (1,100,100,3) the inference speed increases by 3 fold, so basically json.dumps file is too large and thus consuming more time to write and read from memory. I tried ultrajson ie ujson but there wasn't any observable improvement.

trs = img1.tolist()
data = json.dumps({"signature_name": "serving_default", "instances": trs})

As my model architecture requires INT I don't see any other option to serialize the data.

Modifying the input architecture can shift our input requirement of UINT to STRING which can be done by

dl_request = requests.get(IMAGE, stream=True)
jpeg_bytes = base64.b64encode(dl_request.content).decode('utf-8')
predict_request = '{"instances" : [{"b64": "%s"}]}' % jpeg_bytes
response = requests.post(SERVER_URL, data=predict_request)

This fetches good results, but how to convert model input type from INT to STRING?

Any suggestions??

Answer 2

I have been facing the same situation, jason.dumps causes 90% of delay, if the image size is reduced to say (1,100,100,3) the inference speed increases by 3 fold, so basically json.dumps file is too large and thus consuming more time to write and read from memory. I tried ultrajson ie ujson but there wasn't any observable improvement.

trs = img1.tolist()
data = json.dumps({"signature_name": "serving_default", "instances": trs})

As my model architecture requires INT I don't see any other option to serialize the data.

Any suggestions??