Cloud ML model prediction

Question

I have deployed tensorflow saved Model in cloud ML for text classification with the following,

    input_x = graph.get_tensor_by_name('input_x:0')
    keep_prob = graph.get_tensor_by_name('keep_prob:0')
    predictions = graph.get_tensor_by_name('softmax/predictions:0')

feed_dict = {input_x: x_test, batch_size: 8, sequence_length: x_lengths,  keep_prob: 1.0}

Its deployed no error. I have a csv file to predict. --csv file--

"the test is completed"
"the test2 is done"

Getting only errors. How to convert this to json for the model I trained, to batch predict in cloud ML??

saved_model_cli - Info

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['batch_size'] tensor_info:
        dtype: DT_INT32
        shape: ()
        name: batch_size:0
    inputs['input_x'] tensor_info:
        dtype: DT_INT32
        shape: (-1, 25)
        name: input_x:0
    inputs['keep_prob'] tensor_info:
        dtype: DT_FLOAT
        shape: ()
        name: keep_prob:0
    inputs['sequence_length'] tensor_info:
        dtype: DT_INT32
        shape: (-1)
        name: sequence_length:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['predictions'] tensor_info:
        dtype: DT_INT64
        shape: (-1)
        name: softmax/predictions:0
  Method name is: tensorflow/serving/predict

Currently i converted the csv to Json, used to predict:

{"sequence_length": 25, "batch_size": 1, "keep_prob": 1.0, "input_x": [1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 10, 11, 12, 13, 14, 15, 1, 16, 12, 13, 14, 17, 18, 19, 20]}

Exception:

 Exception during running the graph: Cannot feed value of shape (1,) for Tensor u\'keep_prob:0\', which has shape \'()\' (Error code: 2)\n'

Answer 1

This model appears to need several changes to be directly servable. One requirement of the service is that each of the inputs have an unspecified outer dimension which is interpreted as the "batch" dimension. The inputs input_x and sequence_length meet this requirement, but batch_size and keep_prob do not.

The service dynamically builds batches which is why it needs to be variable length. Thus, have an input called batch_size is going to be problematic, since the service doesn't know that it should be setting that input. Instead, it builds up a batch and sends it to TensorFlow. TensorFlow already knows the batch size because it's the value of the outer dimension of inputs such as input_x .

Instead of using batch_size as an input, it's preferable to do something like:

batch_size = tf.shape(input_x)[0]

Although I will note that even the need for that is usually fairly rare in practice. Things usually "just work", because input_x is used in some sort of operation like a matrix multiply or convolution which will handle things fine without explicitly knowing the batch size.

Finally, there is keep_prob , which usually indicates there is a dropout layer in the model. Even though you can hard code this to 1.0, it's usually recommended that you just remove dropout layers altogether for serving. Basically, when you export the model, you actually build a different graph than for training. This is exemplified in this sample .