使用 python 将抓取的数据直接保存到 azure blob 存储

Question

我的需求如下：

I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a.csv or.parquet.

请看示例代码：

response = requests.get('example_url')
        json_response = response.json()
        name = json_response.get('name', 'N/A')
        with open('warsaw.txt', 'w') as f:
            f.write(name)

如何使用Blob Service Client将f直接写入 azure blob 存储？

Answer 1

I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a.csv or.parquet.

我认为您可以使用“附加”方法来满足您的要求：

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, BlobType
try:
    connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
    container_name = "test"
    blob_name = "test.txt"
    data1 = "\n1,2,3"
    data = str.encode(data1)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    print("length of data is "+str(len(data)))
    blob_client.append_block(data,length=len(data))
except:
    connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
    container_name = "test"
    blob_name = "test.txt"
    data1 = "test1,test2,test3"
    data = str.encode(data1)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    print("length of data is "+str(len(data)))
    blob_client.upload_blob(data,blob_type=BlobType.AppendBlob)

通过上述方法，您不必每次都存储整个 blob。

Answer 2

您可以将Azure 存储 Blob 客户端库用于 Python

如何将文件上传到 Azure 存储 Blob 的示例：

from azure.storage.blob import BlobClient

blob = BlobClient.from_connection_string(conn_str="<connection_string>", container_name="my_container", blob_name="my_blob")

with open("./warsaw.txt", "rb") as data:
    blob.upload_blob(data)