sagemaker python sdk (training jobs)是否继承了边缘节点的所有权限？

Question

Working in corporate.network on training a machine learning model. The mlflow tracking works ok with a sagemaker notebook instance but when launching a hyper parameter tuning job from the same sagemaker notebook instance, mlflow tracking will fail:

AlgorithmError: ExecuteUserScriptError: ExitCode 1 ErrorMessage "raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7eff60d845b0>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440, in send resp = conn.urlopen( File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen return self.urlopen( [Previous line repeated 2 more times] File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen retries = retries.increment( File "/opt/conda/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='mlflow.dev.cor p.net', port=80): Max retr

The mlflow tracking uri does have restrictions on corporate access. But I don't see why it blocks the sub-instances launched by sagemaker sdk since the IAM role ARN of the training jobs were inherited from the sagemaker notebook instance. Any solutions on it?

Answer 1

This error isn't related to IAM. The machine you're running this code from doesn't have.network access to: mlflow.dev.corp.net . And apparently this breaks execution.