I'm trying to training TensorFlow model on AWS Sagemaker. I created container with external lib for that (Use Your Own Algorithms or Models with Amazon SageMaker).
we run a training job with TensorFlow API
from sagemaker.tensorflow import TensorFlow
estimator = TensorFlow(
entry_point="entry.py", # entry script
role=role,
framework_version="1.13.0",
py_version='py3',
hyperparameters=hyperparameters,
train_instance_count=1, # "The number of GPUs instances to use"
train_instance_type=train_instance_type,
image_name=my_image
)
estimator.fit({'train': train_s3, 'eval': eval_s3})
and got an error:
09:06:46
2019-07-23 09:06:45,463 INFO - root - running container entrypoint
09:06:46
2019-07-23 09:06:45,463 INFO - root - starting train task
09:06:46
2019-07-23 09:06:45,476 INFO - container_support.training - Training starting
09:06:46
2019-07-23 09:06:45,479 ERROR - container_support.training - uncaught exception during training: No module named 'tf_container'
09:06:46
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/container_support/environment.py", line 136, in load_framework return importlib.import_module('mxnet_container') File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_i
09:06:46
ModuleNotFoundError: No module named 'mxnet_container'
09:06:46
During handling of the above exception, another exception occurred:
09:06:46
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/container_support/training.py", line 35, in start fw = TrainingEnvironment.load_framework() File "/usr/local/lib/python3.6/dist-packages/container_support/environment.py", line 138, in load_framework return importlib.import_module('tf_container') File "/usr/lib/python3.6/importlib/__init__.py", line 126,
09:06:46
ModuleNotFoundError: No module named 'tf_container'
What can I do to solve this issue? how can I debug this case?
I'm guessing that you used your own TF container, not the SageMaker one at https://github.com/aws/sagemaker-tensorflow-container
If that's the case, your container is missing the support code needed to use the TensorFlow estimator ('tf_container' package).
The solution is to start from the SageMaker container, customize it, push it back to ECR and pass the image name to the SageMaker estimator with the 'image_name' parameter.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.