![](/img/trans.png)
[英]How do I retrieve all directory paths from Azure Data Lake Storage Gen2 using Python?
[英]How can I upload a .parquet file from my local machine to Azure Storage Data Lake Gen2?
我的本地计算机中有一组 .parquet 文件,我正在尝试将其上传到 Data Lake Gen2 中的容器。
我不能执行以下操作:
def upload_file_to_directory():
try:
file_system_client = service_client.get_file_system_client(file_system="my-file-system")
directory_client = file_system_client.get_directory_client("my-directory")
file_client = directory_client.create_file("uploaded-file.parquet")
local_file = open("C:\\file-to-upload.parquet",'r')
file_contents = local_file.read()
file_client.append_data(data=file_contents, offset=0, length=len(file_contents))
file_client.flush_data(len(file_contents))
except Exception as e:
print(e)
因为 .read() 函数无法读取 .parquet 文件。
当我尝试这样做时:
def upload_file_to_directory():
file_system_client = service_client.get_file_system_client(file_system="my-file-system")
directory_client = file_system_client.get_directory_client("my-directory")
file_client = directory_client.create_file("uploaded-file.parquet")
file_client.upload_file("C:\\file-to-upload.txt",'r')
我收到以下错误:
AttributeError: 'DataLakeFileClient' object has no attribute 'upload_file'
有什么建议么?
您收到此消息是因为您已导入DataLakeFileClient
模块。 尝试安装DataLakeServiceClient
因为它有upload_file
方法。
pip install DataLakeServiceClient
但是,要读取 .parquet 文件,一种解决方法是使用pandas
。 下面是对我有用的代码。
storage_account_name='<ACCOUNT_NAME>'
storage_account_key='ACCOUNT_KEY'
service_client = DataLakeServiceClient(account_url="{}://{}.dfs.core.windows.net".format(
"https", storage_account_name), credential=storage_account_key)
file_system_client = service_client.get_file_system_client(file_system="container")
directory_client = file_system_client.get_directory_client(directory="directory")
file_client = directory_client.create_file("uploaded-file.parquet")
local_file = pd.read_parquet("<YOUR_FILE_NAME>.parquet")
df = pd.DataFrame(local_file).to_parquet()
file_client.upload_data(data=df,overwrite=True) #Either of the lines works
#file_client.append_data(data=df, offset=0, length=len(df))
file_client.flush_data(len(df))
并且您可能需要导入DataLakeFileClient
库才能使其工作:
from azure.storage.filedatalake import DataLakeServiceClient
import pandas as pd
结果:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.