无法使用 python 从 Azure 数据湖中加载 tensorflow keras model

Question

I have a model that I saved inside a project_name in a container in azure data lake. I am having issues when loading a tensorflow model from databricks. Everything is already working fine as I tested out in jupyter notebook previously. I have to migrate the code in databricks.

This is a code I ran.

model = tf.keras.models.load_model("abfss://dev@abcproddatalake.dfs.core.windows.net/hkay/project_name/model/keras2.tf", compile=False)

This is the error I'm getting.

UnimplementedError                        Traceback (most recent call last)
<command-3129204037083358> in <module>
      3 
      4 ## Loading a model
----> 5 model = tf.keras.models.load_model("abfss://dev@abcproddatalake.dfs.core.windows.net/hkay/project_name/model/keras2.tf", compile=False)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-03228642-df50-44d1-8e0e-f760ea5a0429/lib/python3.8/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     68             # To get the full stack trace, call:
     69             # `tf.debugging.disable_traceback_filtering()`
---> 70             raise e.with_traceback(filtered_tb) from None
     71         finally:
     72             del filtered_tb

/local_disk0/.ephemeral_nfs/envs/pythonEnv-03228642-df50-44d1-8e0e-f760ea5a0429/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py in file_exists_v2(path)
    288   """
    289   try:
--> 290     _pywrap_file_io.FileExists(compat.path_to_bytes(path))
    291   except errors.NotFoundError:
    292     return False

UnimplementedError: File system scheme 'abfss' not implemented (file: 'abfss://dev@abcproddatalake.dfs.core.windows.net/hkay/project_name/model/keras2.tf')

It was running fine when I ran it in jupyter notebook. The model was saved locally when I was using jupyter.

The connection is working because I tested out the connection through reading the files from the path. The size of this model is 1.6 GB. I am not sure why it is not working. Anyone has any idea?

Answer 1

Unfortunately, Keras doesn't understand URIs. it is designed to work only with local files,so you need to use local path saving or loading data.

Try to use dbutils.fs.cp it will copy data from Storage URL abfss://dev@axxx to local path.

Copy file into /tmp/model.tf and load it.

#Setup storage configuration
spark.conf.set("fs.azure.account.key.<storage_account_name>.blob.core.windows.net","<access_key>")

dbutils.fs.cp("abfss://dev@abcproddatalake.dfs.core.windows.net/hkay/project_name/model/keras2.tf","/tmp/Demo_model.tf")

you can check weather model copied or not using below code.

display(dbutils.fs.ls("/tmp/Demo_model.tf"))

Loading model:

from tensorflow import keras
model = keras.models.load_model("/tmp/Demo_model.tf")