尝试使用 tf.saved_model.load() 加载 tensorflow 时出现“读取的字节数少于请求的字节数”错误

Question

I've been trying to run a web server locally that my colleague created which contains a deployed model that processes object detection requests. There've been a lot of compatibility issues because he's on Windows and I'm on MacOS.

I've been able to resolve all the compatibility issues with the dependencies and get the web server running, but I've been running into this issue when trying to load the object detection model from the saved_model directory. The structure of the directory is:
saved_model/
variables/
variables.data-00000-of-00001
variables.index
saved_model.pb

I created a test script to try isolate the problem, which looks as follows:

import tensorflow as tf

model = tf.saved_model.load("./saved_model")

# print model summary
print("loaded model")
print(model.summary())

When I run the script, I get the following error, which is the same error that I get when running the web server.

Traceback (most recent call last): File "/Users/alexandrospouroullis/programming-projects/elevat3d/backend/workspace/training_demo/exported-models/my_model_1024/testFile.py", line 4, in model = tf.saved_model.load("./saved_model") File "/Users/alexandrospouroullis/opt/miniconda3/envs/elevat3d-api/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 782, in load result = load_partial(export_dir, None, tags, options)["root"] File "/Users/alexandrospouroullis/opt/miniconda3/envs/elevat3d-api/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 912, in load_partial loader = Loader(object_graph_proto, saved_model_proto, export_dir, File "/Users/alexandrospouroullis/opt/miniconda3/envs/elevat3d-api/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 189, in init self._restore_checkpoint() File "/Users/alexandrospouroullis/opt/miniconda3/envs/elevat3d-api/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 507, in _restore_checkpoint load_status = saver.restore(variables_path, self._checkpoint_options) File "/Users/alexandrospouroullis/opt/miniconda3/envs/elevat3d-api/lib/python3.10/site-packages/tensorflow/python/training/tracking/util.py", line 1430, in restore object_graph_string = reader.get_tensor(base.OBJECT_GRAPH_PROTO_KEY) File "/Users/alexandrospouroullis/opt/miniconda3/envs/elevat3d-api/lib/python3.10/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 66, in get_tensor return CheckpointReader.CheckpointReader_GetTensor( IndexError: Read less bytes than requested

I understand that things like Docker were made for this; I actually created a docker-compose.yaml file that I managed to get up and running as well, but it faced this exact same issue.

I've looked around on the web, but this doesn't seem to be a common issue and no remedy, as far as I know, exists.

What could be going wrong?

Answer 1

There are two things you will have to check:

1. Make sure your weights file(under the variables folder) is properly loaded in your docker environment(check the size of the file). At times, the files is not properly loaded and in just 2 Kb whereas the actual file will be atleast 0.5 GB and upwards.
2. Enough memory while loading the model. TF models take up considerable infra juice and atleast 16GB RAM is recommended. I faced a similar issue. Mine was because of a corrupt weights file - point1.