Python: azure-storage-blob takes 6 min to download a 20MB file to local

Question

I have to download .parqeat file from Azure storage and have to analyze the Data. So I started using python codes with azure-storage-blob library and Pandas to store DataFrames and flask to create API on response. But now file size in the Azure storage is getting increased up to 30MB(a day file) and as tested it takes 6min to download 20MB file into my local which I can't afford because I have to analyse files of each days in a month, which will have 20 MB each. That's almost 30 20MB size. Any suggestion?

Answer 1

Did you find your answer? Without a code snippet it would be hard to diagnos your problem. Azure Blob Storage supports read in parallel. If you have BatchServiceClient service that would be straightforward. Otherwise, try Multiprocessing module. Kr.