使用 python 将文件从 azure 数据湖 Gen 1 移动到临时目录

Question

如何使用 python 将文件从 azure 数据湖 Gen 1 移动到临时目录？ 我需要在 ADL 中找到每个文件，将其移动到临时目录以解密并读取它。

我在用

azure.datalake.store import core
core.AzureDLFileSystem(token, 'store_name')

#create a temp directory
 with tempfile.TemporaryDirectory() as tmpdirname:

Answer 1

关于这个问题，请参考以下代码

from azure.datalake.store import core, lib, multithreaded

##auth
adlCreds = lib.auth(tenant_id = tenant,
                client_secret = client_secret,
                client_id = client_id,
                resource = RESOURCE)
subscriptionId = 'FILL-IN-HERE'
adlsAccountName = 'FILL-IN-HERE'

## Create a filesystem client object
adlsFileSystemClient = core.AzureDLFileSystem(adlCreds, store_name=adlsAccountName)


## Download a file
multithread.ADLDownloader(adlsFileSystemClient, lpath='<local path>', rpath='<path on adls>', nthreads=64, overwrite=True, chunksize=4194304,buffersize=4194304, blocksize=4194304)

##Move file between locations on ADL
adlsFileSystemClient.mv('<Source Path>','Destination path')

有关更多详细信息，请参阅此处和此处