對 TensorFlow 服務的 RaggedTensor 請求失敗

Question

我創建了一個使用 RaggedTensors 的 TensorFlow model。 Model 工作正常，當調用model.predict時，我得到了預期的結果。

input = tf.ragged.constant([[[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
                                [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963], 
                                [-0.9984272718429565, -0.942145586013794, 2.48842453956604, -1.6836735010147095, -0.6360141634941101, -1.5785763263702393, -1.900200605392456, -0.49169546365737915, -0.6656315326690674, -0.6583622694015503, -0.5273991227149963], 
]])
model.predict(input)

>> array([[0.5138151 , 0.3277698 , 0.26122513]], dtype=float32)

我已將 model 部署到 TensorFlow 服務服務器，並使用以下代碼調用：

import json
import requests
headers = {"content-type": "application/json"}
data = json.dumps({"instances":[
    [-1.3523329846758267, ... more data ],
    [-1.3523329846758267, ... more data ],
    [-1.3523329846758267, ... more data ],
    [-1.3523329846758267, ... more data ,
    [-1.3523329846758267, ... more data ],
    [-1.3523329846758267, ... more data ],
    [-1.3523329846758267, ... more data ],
    [-1.3523329846758267, ... more data })
json_response = requests.post('http://localhost:8501/v1/models/fashion_model:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)

但后來我收到以下錯誤：

"instances is a plain list, but expecting list of objects as multiple input tensors required as per tensorinfo_map"

我的 model 描述：

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['args_0'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 11)
        name: serving_default_args_0:0
    inputs['args_0_1'] tensor_info:
        dtype: DT_INT64
        shape: (-1)
        name: serving_default_args_0_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['dense_2'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 3)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict
WARNING: Logging before flag parsing goes to stderr.
W0124 09:33:16.365564 140189730998144 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling __init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Defined Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

我錯過了什么？

更新：檢查saved_model_cli output 后，我懷疑我應該像下面這樣將請求作為 object 發送，但我不確定輸入...

{
  "instances": [
    {
      "args_0": nested-list ?,
      "args_0_1": ???
    }
  ]
}

更新 2用於測試此場景的Colab ，Colab 中包含下載 model 的鏈接。

更新 3：

正如@Niteya Shah 所建議的那樣，我將 API 稱為：

data = json.dumps({
 "inputs": {
   "args_0": [[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
              [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963]],
   "args_0_1": [1, 2]  #Please Check what inputs come here ?
  }
})

並得到了結果（最后：）：

{'outputs': [[0.466771603, 0.455221593, 0.581544757]]}

然后使用相同的數據調用 model，如下所示：

import numpy as np
input = tf.ragged.constant([[
                            [-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
                            [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963]
]])
model.predict(input)

並得到了不同的結果：

array([[0.4817084 , 0.3649785 , 0.01603118]], dtype=float32)

我想我還是不在。

Answer 1

https://www.tensorflow.org/tfx/serving/api_rest#predict_api

我認為您需要使用 REST API 中推薦的列格式而不是行格式，因為您的第 0 個輸入的尺寸不匹配。 這意味着您將不得不使用輸入而不是實例。 由於您還有多個輸入，因此您還必須將其作為命名輸入提及。

示例數據請求可能如下所示

data = json.dumps({
 "inputs": {
   "args_0": [[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
              [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963]],
   "args_0_1": [10, 11]  #Substitute this with the correct row partition values. 
  }
})

編輯：

我從這里讀到了 Ragged 張量，似乎第二個輸入可能是行分區。 我在文檔中找不到關於通常使用哪種行分區樣式的信息，所以我使用的是行長度方法。 幸運的是 TensorFlow ragged 提供了為我們執行此操作的方法。 使用values和row_splits屬性來訪問它們。 那應該行得通。

Answer 2

其他人可能會從中受益，因為我花了一段時間才縫合在一起：

1. 在不規則張量上訓練玩具 LSTM model。
2. 將其加載到 TensorFlow 服務中。
3. 使用序列化的不規則張量發出預測請求。

如果有人知道如何重命名“args_0”和“args_0_1”，請添加。 相關 Git 問題： https://github.com/tensorflow/tensorflow/issues/37226

構建和保存 Model

TensorFlow 版本：2.9.1 Python 版本：3.8.12

# Task: predict whether each sentence is a question or not.
sentences = tf.constant(
    ['What makes you think she is a witch?',
     'She turned me into a newt.',
     'A newt?',
     'Well, I got better.'])
is_question = tf.constant([True, False, True, False])

# Preprocess the input strings.
hash_buckets = 1000
words = tf.strings.split(sentences, ' ')
hashed_words = tf.strings.to_hash_bucket_fast(words, hash_buckets)


# Build the Keras model.
keras_model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=[None], dtype=tf.int64, ragged=True),
    tf.keras.layers.Embedding(hash_buckets, 16),
    tf.keras.layers.LSTM(32, use_bias=False),
    tf.keras.layers.Dense(32),
    tf.keras.layers.Activation(tf.nn.relu),
    tf.keras.layers.Dense(1)
])

keras_model.compile(loss='binary_crossentropy', optimizer='rmsprop')
keras_model.fit(hashed_words, is_question, epochs=5)

print(keras_model.predict(hashed_words))

keras_module_path = "/home/ec2-user/SageMaker/keras-toy-lstm/1"
tf.keras.Model.save(keras_model, keras_module_path)

從 Model 加載和推斷

將 model 裝入 TensorFlow 服務容器

docker run -t --rm -p 8501:8501 -v "/home/ec2-user/SageMaker/keras-toy-lstm/:/models/keras-model" -e MODEL_NAME=keras-model  tensorflow/serving 

import requests
import json 

payload = {"args_0": [940, 203, 668, 638], 
            "args_0_1": [0, 4]}
headers = {"content-type": "application/json"}
data = json.dumps({"inputs":payload})

r = requests.post('http://localhost:8501/v1/models/keras-model:predict', data=data, headers=headers)
r.text

已保存模型CLI Output

(tensorflow2_p38) sh-4.2$ saved_model_cli show --dir /tmp/tmpgp0loz1v/ --all

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['args_0'] tensor_info:
        dtype: DT_INT64
        shape: (-1)
        name: serving_default_args_0:0
    inputs['args_0_1'] tensor_info:
        dtype: DT_INT64
        shape: (-1)
        name: serving_default_args_0_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['dense_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict

Concrete Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None