如何使用Python将Azure Blob存储中的大型JSON文件拆分为每个记录的单个文件？

Question

I want to be able to split some large JSON files in blob storage (~1GB each) into individual files (one file per record)

I have tried using get_blob_to_stream from the Azure Python SDK, but am getting the following error:

AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

To test, I've just been printing the text that has been downloaded from the blob, and haven't yet tried writing back to individual JSON files

with BytesIO() as document:
    block_blob_service = BlockBlobService(account_name=STORAGE_ACCOUNT_NAME, account_key=STORAGE_ACCOUNT_KEY)
    block_blob_service.get_blob_to_stream(container_name=CONTAINER_NAME, blob_name=BLOB_ID, stream=document)

    print(document.getvalue())

Interestingly, when I limit the size of the blob information that I'm downloading, the error message doesn't appear, and I can get some information out:

with BytesIO() as document:
    block_blob_service = BlockBlobService(account_name=STORAGE_ACCOUNT_NAME, account_key=STORAGE_ACCOUNT_KEY)
    block_blob_service.get_blob_to_stream(container_name=CONTAINER_NAME, blob_name=BLOB_ID, stream=document, start_range=0, end_range=100000)

    print(document.getvalue())

Does anyone know what is going on here, or have any better approaches to splitting a large JSON out?

Thanks!

Answer 1

This Error message "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature" is you get usually when the header is not formed correctly. you get following when you get this error:

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>AuthenticationFailed</Code>
    <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:096c6d73-f01e-0054-6816-e8eaed000000
Time:2019-03-31T23:08:43.6593937Z</Message>
    <AuthenticationErrorDetail>Authentication scheme Bearer is not supported in this version.</AuthenticationErrorDetail>
</Error>

and the solution to resolve this is to add below header:

x-ms-version: 2017-11-09

But since you are saying that it is working when you limit the size which means you have to write your code using chunk approach. Here is something you can try.

import io
import datetime
from azure.storage.blob import BlockBlobService

acc_name = 'myaccount'
acc_key = 'my key'
container = 'storeai'
blob = "orderingai2.csv"

block_blob_service = BlockBlobService(account_name=acc_name, account_key=acc_key)
props = block_blob_service.get_blob_properties(container, blob)
blob_size = int(props.properties.content_length)
index = 0
chunk_size =  104,858 # = 0.1meg don't make this to big or you will get memory error
output = io.BytesIO()


def worker(data):
    print(data)


while index < blob_size:
    now_chunk = datetime.datetime.now()
    block_blob_service.get_blob_to_stream(container, blob, stream=output, start_range=index, end_range=index + chunk_size - 1, max_connections=50)
    if output is None:
        continue
    output.seek(index)
    data = output.read()
    length = len(data)
    index += length
    if length > 0:
        worker(data)
        if length < chunk_size:
          break
    else:
      break

Hope it helps.