有没有办法从 triton 推理服务器获取 config.pbtxt 文件

Question

Recently, I have come across a solution of the triton serving config file disable flag "--strict-model-config=false" while running the inferencing server. This would enable to create its own config file while loading the model from the model repository.

sudo docker run --rm --net=host -p 8000:8000 -p 8001:8001 -p 8002:8002 \
-v /home/rajesh/custom_repository:/models nvcr.io/nvidia/tritonserver:22.06-py3 \
tritonserver --model-repository=/models --strict-model-config=false

I would like to get the generated config file from the triton inferencing server since we can play around with the batch config and other parameters. Is there a way to get the inbuilt generated config.pbtxt file for the models I have loaded in the server so that I can play around the batch size and other parameters.

Answer 1

As per the reference from the below link, the loaded model configuration can be found after loading the model repository into the triton server using the below curl command.

https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md#minimal-model-configuration

command:

curl localhost:8000/v2/models/<model_name>/config

Answer 2

The above answer which the uses curl command would return the json response.

If the results should be in the protobuf format, try loading the model using triton inferencing server with strict model config as false and fetch the results by using the below python script which would return the results in necessary protobuf format. Use this to get the format of the model and edit it easily as per the needs in config pbtxt file instead of cnoverting json to protobuf results.

import tritonclient.grpc as grpcclient

triton_client = grpcclient.InferenceServerClient(url=<triton_server_url>)

model_config = triton_client.get_model_config(model_name=<model_name>, model_version=<model_version>)