ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::astor-0.7.1-py_0'

Question

I am trying to follow a Python tutorial and I have been able to execute almost everything, until the point of Deploying an endpoint to Azure with python.

In order to give some context I have uploaded the scripts to my git account: https://github.com/levalencia/MLTutorial

File 1 and 2 Work perfectly fine

However the following section in File 3 fails:

%%time
from azureml.core.webservice import Webservice
from azureml.core.model import InferenceConfig

inference_config = InferenceConfig(runtime= "python", 
                                   entry_script="score.py",
                                   conda_file="myenv.yml")

service = Model.deploy(workspace=ws, 
                       name='keras-mnist-svc2', 
                       models=[amlModel], 
                       inference_config=inference_config, 
                       deployment_config=aciconfig)

service.wait_for_deployment(show_output=True)

with below error:

ERROR - Service deployment polling reached non-successful terminal state, current service state: Transitioning
Operation ID: 8353cad2-4218-450a-a03b-df418725acb1
More information can be found here: https://machinelearnin1143382465.blob.core.windows.net/azureml/ImageLogs/8353cad2-4218-450a-a03b-df418725acb1/build.log?sv=2018-03-28&sr=b&sig=UKzefxIrm3l7OsXxj%2FT4RsvUfAuhuaBwaz2P4mJu7vY%3D&st=2020-03-11T12%3A23%3A33Z&se=2020-03-11T20%3A28%3A33Z&sp=r
Error:
{
  "code": "EnvironmentBuildFailed",
  "statusCode": 400,
  "message": "Failed Building the Environment."
}

ERROR - Service deployment polling reached non-successful terminal state, current service state: Transitioning
Operation ID: 8353cad2-4218-450a-a03b-df418725acb1
More information can be found here: https://machinelearnin1143382465.blob.core.windows.net/azureml/ImageLogs/8353cad2-4218-450a-a03b-df418725acb1/build.log?sv=2018-03-28&sr=b&sig=UKzefxIrm3l7OsXxj%2FT4RsvUfAuhuaBwaz2P4mJu7vY%3D&st=2020-03-11T12%3A23%3A33Z&se=2020-03-11T20%3A28%3A33Z&sp=r
Error:
{
  "code": "EnvironmentBuildFailed",
  "statusCode": 400,
  "message": "Failed Building the Environment."
}

When I download the logs, I got this:

wheel-0.34.2         | 24 KB     |            |   0% [0m[91m
wheel-0.34.2         | 24 KB     | ########## | 100% [0m
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... failed
[91m
ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::astor-0.7.1-py_0'.
FileNotFoundError(2, "No such file or directory: '/azureml-envs/azureml_6abde325a12ccdba9b5ba76900b99b56/bin/python3.6'")
Attempting to roll back.

[0mRolling back transaction: ...working... done
[91m
FileNotFoundError(2, "No such file or directory: '/azureml-envs/azureml_6abde325a12ccdba9b5ba76900b99b56/bin/python3.6'")


[0mThe command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_6abde325a12ccdba9b5ba76900b99b56 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig' returned a non-zero code: 1
2020/03/11 12:28:11 Container failed during run: acb_step_0. No retries remaining.
failed to run step ID: acb_step_0: exit status 1

Run ID: cb3 failed after 2m21s. Error: failed during run, err: exit status 1

Update 1:

I tried to run: conda list --name base conda

inside the notebook and I got this:

 # packages in environment at /anaconda:
    #
    # Name                    Version                   Build  Channel
    _anaconda_depends         2019.03                  py37_0  
    anaconda                  custom                   py37_1  
    anaconda-client           1.7.2                    py37_0  
    anaconda-navigator        1.9.6                    py37_0  
    anaconda-project          0.8.4                      py_0  
    conda                     4.8.2                    py37_0  
    conda-build               3.17.6                   py37_0  
    conda-env                 2.6.0                         1  
    conda-package-handling    1.6.0            py37h7b6447c_0  
    conda-verify              3.1.1                    py37_0  

    Note: you may need to restart the kernel to use updated packages.

However in the deployment log I got this:

Solving environment: ...working... 
done
[91m

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.11
  latest version: 4.8.2

Please update conda by running

    $ conda update -n base -c defaults conda

Answer 1

Unfortunately there seems to an issue with this version of Conda (4.5.11). To complete this task in the tutorial, you can just update the dependency for Tensorflow and Keras to be from pip and not conda . There are reasons why this is less than ideal for a production environment. The Azure ML documentation states :

"If your dependency is available through both Conda and pip (from PyPi), use the Conda version, as Conda packages typically come with pre-built binaries that make installation more reliable."

In this case though, if you update the following code block:

from azureml.core.conda_dependencies import CondaDependencies 

myenv = CondaDependencies()
myenv.add_conda_package("tensorflow")
myenv.add_conda_package("keras")

with open("myenv.yml","w") as f:
    f.write(myenv.serialize_to_string())

# Review environment file
with open("myenv.yml","r") as f:
    print(f.read())

To be the following:

from azureml.core.conda_dependencies import CondaDependencies 

myenv = CondaDependencies()
myenv.add_pip_package("tensorflow==2.0.0")
myenv.add_pip_package("azureml-defaults")
myenv.add_pip_package("keras")

with open("myenv.yml", "w") as f:
    f.write(myenv.serialize_to_string())

with open("myenv.yml", "r") as f:
    print(f.read())

The tutorial should be able to be completed. Let me know if any of this does not work for you once this update has been made.

I have also reported this issue to Microsoft (in regards to the Conda version).