从 azure blob 并行读取多个文件

Question

I'm looking to read a bunch on small files from an azure blob, this can be in the order of 1k-100k files summing up few 1TB in total. I have to process this files in python, the processing it self is not heavy, but reading the files from the blob it does takes time. One other constrain of this, is that new files are been writeen while I process the first one.

I'm looking for options to do this, is it possible to use dask to read several files from the blob in parallel? or is it possible to transfer and load over 1tb per hour inside the azure network?

Answer 1

Well, you have a couple of options to achieve parallelism here:

Multi-threading:

Below uses uses ThreadPool class in Python to download and process files in parallel from Azure storage. Note: using v12 storage sdk

import os
from multiprocessing.pool import ThreadPool
from azure.storage.blob import BlobServiceClient

STORAGE_CONNECTION_STRING = "REPLACE_THIS"
BLOB_CONTAINER = "myfiles"

class AzureBlobProcessor:
  def __init__(self): 
    # Initialize client
    self.blob_service_client =  BlobServiceClient.from_connection_string(STORAGE_CONNECTION_STRING)
    self.blob_container = self.blob_service_client.get_container_client(BLOB_CONTAINER)
 
  def process_all_blobs_in_container(self):
    # get a list of blobs
    blobs = self.blob_container.list_blobs()
    result = self.execute(blobs)
 
  def execute(self, blobs):
    # Just sample number of threads as 10
    with ThreadPool(processes=int(10)) as pool:
     return pool.map(self.download_and_process_blob, blobs)
 
  def download_and_process_blob(self,blob):
    file_name = blob.name
    
    # below is just sample which reads bytes, update to variant you need
    bytes = self.blob_container.get_blob_client(blob).download_blob().readall()
 
    # processing logic goes here :)

    return file_name
 
# caller code
azure_blob_processor = AzureBlobProcessor()
azure_blob_processor.process_all_blobs_in_container()

You can also look at dask remote data read . Check https://github.com/dask/adlfs

To use the Gen1 filesystem:

import dask.dataframe as dd

storage_options={'tenant_id': TENANT_ID, 'client_id': CLIENT_ID, 'client_secret': CLIENT_SECRET}

dd.read_csv('adl://{STORE_NAME}/{FOLDER}/*.csv', storage_options=storage_options)

To use the Gen2 filesystem you can use the protocol abfs or az :

import dask.dataframe as dd

storage_options={'account_name': ACCOUNT_NAME, 'account_key': ACCOUNT_KEY}

ddf = dd.read_csv('abfs://{CONTAINER}/{FOLDER}/*.csv', storage_options=storage_options)
ddf = dd.read_parquet('az://{CONTAINER}/folder.parquet', storage_options=storage_options)

To read from a public storage blob you are required to specify the 'account_name' . For example, you can access NYC Taxi & Limousine Commission as:

storage_options = {'account_name': 'azureopendatastorage'}
ddf = dd.read_parquet('az://nyctlc/green/puYear=2019/puMonth=*/*.parquet', storage_options=storage_options)

Achieve parallelism leveraging Azure PaaS:

Well, you have multiple options in this path.

• Azure Batch:Tutorial: Run a parallel workload with Azure Batch using the Python API
• Azure Function with Blob Event Grid Trigger: Azure Event Grid bindings for Azure Functions

---

Lastly, I suggest you a deep dive at Performance and scalability checklist for Blob storage to ensure you are within limit of Azure storage account data transfer. Also Scalability and performance targets for standard storage accounts . Looking at your 1 tb per hour requirement, it seems under limit if you convert the gbps from the above doc.