Azure Python download storage blob returns 'The condition specified using HTTP conditional header(s) is not met.'

Question

With the following code:

import os
from azure.identity import (
    ClientSecretCredential
)

# Import the client object from the Azure library
from azure.storage.blob import BlobClient

t_id = "<tenant_id>"
c_id = "<client_id>"
s_acct = "<storage_account_name>"
s_acct_url = "%s.blob.core.windows.net" % s_acct
sek = "<client_sekret>"

print("+ Setup credentials.")
credential = ClientSecretCredential(t_id, c_id, sek)
print("+ Setup Blob Client")
bobc = BlobClient(s_acct_url, <container_name>, <blob_name>,
                  credential=credential)
print("+ Setup streamer")
ssd = bobc.download_blob()
print("+ Get properties")
print(ssd.get_blob_properties())

But I get the following error:

$ python azdown.py
+ Setting up stream
+
+ Download stream:
+    Size: 136365212160
Traceback (most recent call last):
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 192, in _download_chunk
    _, response = self.client.download(
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_generated\operations\_blob_operations.py", line 179, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The condition specified using HTTP conditional header(s) is not met.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "azdown.py", line 29, in <module>
    download_stream.download_to_stream(my_blob)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 579, in download_to_stream
    self.readinto(stream)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 561, in readinto
    downloader.process_chunk(chunk)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 125, in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 201, in _download_chunk
    process_storage_error(error)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_shared\response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ResourceModifiedError: The condition specified using HTTP conditional header(s) is not met.
RequestId:<request id>
Time:2021-01-10T01:23:24.8981731Z
ErrorCode:ConditionNotMet
Error:None

I looked at [1], but I am using download_to_stream(). Then I tried [2] but my storage is a gen1 container. There was a link that it had something to do with the container's network permissions but the container is set to allow all to download. The problem is, it does download up to a certain point (atm, 1.2G out of 120G) and then it chokes. So it isn't a permission issue but some sort of race condition.

[1] - https://social.msdn.microsoft.com/Forums/azure/en-US/3b4df832-3340-4415-8d93-d71662e1c540/azure-blob-the-condition-specified-using-http-conditional-headers-is-not-met

[2] - https://forums.databricks.com/questions/20415/intermittent-http-error-when-loading-files-from-ad.html

[edit]

I also tried using the connection_string method:

credential = DefaultAzureCredential()

    blob_url = "DefaultEndpointsProtocol=https;AccountName=<storage_acct>;" + \
               "AccountKey=<account_key>;" + \
               "EndpointSuffix=core.windows.net"

    # Create the client object using the storage URL and the credential
    blob_client = BlobClient.from_connection_string(
        blob_url,
        container_name=<container_name>,
        blob_name=<blob_item>)

    if download_now:
        with open(<blob_item>, "wb") as my_blob:
            print("+ Setting up stream")
            download_stream = blob_client.download_blob()
            print("+")
            print("+ Download stream:")
            print("+    Size: %d" % download_stream.size)
            download_stream.download_to_stream(my_blob)

    else:
        print(blob_client.get_blob_properties())

I get the same kind of error even after it has streamed a gig or so.

Error:

Traceback (most recent call last):
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 192, in _download_chunk
    _, response = self.client.download(
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_generated\operations\_blob_operations.py", line 179, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The condition specified using HTTP conditional header(s) is not met.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "azuredl.py", line 52, in <module>
    download_stream.download_to_stream(my_blob)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 579, in download_to_stream
    self.readinto(stream)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 561, in readinto
    downloader.process_chunk(chunk)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 125, in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 201, in _download_chunk
    process_storage_error(error)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_shared\response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ResourceModifiedError: The condition specified using HTTP conditional header(s) is not met.
RequestId:<request id>
Time:2021-01-10T07:43:24.4927426Z
ErrorCode:ConditionNotMet
Error:None

Might anyone have any ideas?

Answer 1

I can repro this issue when I make some updates(such as change content or adding some metadata) on the file that I am downloading:

This issue is due to once the blob gets updated, its Etag changes at the same time. Which causes this issue. Details see this doc .

To solve this issue, you can require a Lise for this blob to add a lock so that this blob will not be edit or delete while downloading.

Try code below:

blob_client = BlobClient.from_connection_string(conn_str='', container_name='',blob_name='')

#require lease that never expires
lease = blob_client.acquire_lease(lease_duration=-1)

with open("<some path>", "wb") as my_blob:
            print("+ Setting up stream")
            download_stream = blob_client.download_blob()
            print("+")
            print("+ Download stream:")
            print("+    Size: %d" % download_stream.size)
            download_stream.download_to_stream(my_blob)

#break lease after download
lease.break_lease()