如何从 memory 中的 azure 存储中读取大型 gzip csv 文件 Z945F3FC4429568A73B6ZF5？

Question

I'm trying to transform large gzip csv files(>3 Gigs) from azure storage blob by loading it to pandas dataframe in AWS Lambda function.

Local machine with 16 gigs is able to process my files but it gives memory error "errorType": "MemoryError", while executing in aws lambda, since lambda has max memory is 3 Gb.

Is there any way of reading and processing those large data in memory in aws lambda. Just to mention I tried stream way as well but no luck.

Below is my sample code-

from azure.storage.blob import *
import pandas as pd
import gzip

blob = BlobClient(account_url="https://" + SOURCE_ACCOUNT + ".blob.core.windows.net",
                          container_name=SOURCE_CONTAINER,
                          blob_name=blob_name,
                          credential=SOURCE_ACCT_KEY)
    
data = blob.download_blob()
data = data.readall()
    
unzipped_bytes = gzip.decompress(data)
unzipped_str = unzipped_bytes.decode("utf-8")
df = pd.read_csv(StringIO(unzipped_str), usecols=req_columns,low_memory=False)

Answer 1

Try using chunks and read data by n-rows per one time:

for chunk in pd.read_csv("large.csv", chunksize=10_000):
    print(chunk)

Although, I'm not sure about compressed files. In worst case you can uncompress data and then read by chunks.