使用 urllib3 分段上传 - 值错误

Question

I'm writing a script to upload a large file (50GB) to our service at work which utilizes minio.

config = TransferConfig(multipart_threshold=1024 * 100 * 1024, max_concurrency=10,
                        multipart_chunksize=1024 * 100 * 1024, use_threads=True)

session = boto3.Session(
    aws_access_key_id=upload_data['access_key_id'],
    aws_secret_access_key=upload_data['secret_access_key'],
    aws_session_token=upload_data['session_token'],
    region_name=upload_data['region']
)

client = session.client('s3', endpoint_url='https://maurice-storage.vdoo.team/', verify=False)
client.upload_file(
    image_file,
    upload_data['bucket'],
    upload_data['key'],
    ExtraArgs={'Metadata': {
        'name': os.path.basename(image_file),
        'size': str(os.stat(image_file).st_size),
    }},
    Config = config,
    Callback=ProgressPercentage(image_file)
)

I've set the upload to upload chunks of 100 MB (total 500 chunks) as the file is 50GB.

However, once the upload reaches 100%, a ValueError exception is thrown:

File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\response.py", line 696, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

I know why it happens - it tries to call int with an empty binary value (b''), but i'm not sure why that value is there.

Any idea why that could happen?

Answer 1

Looks like an issue with the server where after advertising chunked encoding and sending all the chunks, it is not sending the final zero length chunk to indicate the end of the chunked response.

More info here and especially in this comment .