将解压缩的文件写入从Web服务器获取的磁盘

Question

I can get a file that has content-encoding as gzip .

So does that mean that the server is storing it as compressed file or it is also true for files stored as compressed zip or 7z files too?

and if so (where durl is a zip file)

>>> durl = 'https://db.tt/Kq0byWzW'
>>> dresp = requests.get(durl, allow_redirects=True, stream=True)
>>> dresp.headers['content-encoding']
'gzip'

>>> r = requests.get(durl, stream=True)
>>> data = r.raw.read(decode_content=True)

but data is coming out to be empty while I want to extract the zip file to disk on the go !!

Answer 1

You need the content from the requests file to write it. Confirmed working:

import requests
durl = 'https://db.tt/Kq0byWzW'
dresp = requests.get(durl, allow_redirects=True, stream=True)
dresp.headers['content-encoding']

file = open('test.html', 'w')
file.write(dresp.text)

Answer 2

So first of all durl is not a zip file, it is a drop box landing page. So what you are looking at is HTML which is being sent using gzip encoding. If you where to decode the data from the raw socket using gzip you would simply get the HTML. So the use of raw is really just hiding that you accidentally go an other file than the one you thought.

Based on https://plus.google.com/u/0/100262946444188999467/posts/VsxftxQnRam where you ask

Does anyone has any idea about writing compressed file directy to disk to decompressed state?

I take it you are really trying to fetch a zip and decompress it directly to a directory without first storing it. To do this you need to use https://docs.python.org/2/library/zipfile.html

Though at this point the problem becomes that the response from requests isn't actually seekable, which zipfile requires in order to work (one of the first things it will do is seek to the end of the file to determine how long it is).

To get around this you need to wrap the response in a file like object. Personally I would recommend using tempfile.SpooledTemporaryFile with a max size set. This way your code would switch to writing things to disk if the file was bigger than you expected.

import requests
import tempfile
import zipfile

KB = 1<<10
MB = 1<<20

url = '...' # Set url to the download link.

resp = requests.get(url, stream=True)
with tmp as tempfile.SpooledTemporaryFile(max_size=500*MB):
    for chunk in resp.iter_content(4*KB):
        tmp.write(chunk)
    archive = zipfile.ZipFile(tmp)
    archive.extractall(path)

Same code using io.BytesIO :

resp = requests.get(url, stream=True)
tmp = io.BytesIO()
for chunk in resp.iter_content(4*KB):
    tmp.write(chunk)
archive = zipfile.ZipFile(tmp)
archive.extractall(path)

Answer 3

You have to differentiate between content-encoding (not to be confused with transfer-encoding ) and content-type .

The gist of it is that content-type is the media-type (the real file-type) of the resource you are trying to get. And content-encoding is any kind of modification applied to it before sending it to the client.

So let's assume you'd like to get a resource named "foo.txt". It will probably have a content-type of text/plain .In addition to that, the data can be modified when sending over the wire. This is the content-encoding . So, with the above example, you can have a content-type of text/plain and a content-encoding of gzip . This means that before the server sends the file out onto the wire, it will compress it using gzip on the fly. So the only bytes which traverse the net are zipped. Not the raw-bytes of the original file ( foo.txt ).

It is the job of the client to process these headers accordingly.

Now, I am not 100% sure if requests , or the underlying python libs do this but chances are they do. If not, Python ships with a default gzip library , so you could do it on your own without a problem.

With the above in mind, to respond to your question: No, having a "content-encoding" of gzip does not mean that the remote resource is a zip-file. The field containing that information is content-type (based on your question this has probably a value of application/zip or application/x-7z-compressed depending of actual compression algorithm used).

If you cannot determine the real file-type based on the content-type field (f.ex. if it is application/octet-stream ), you could just save the file to disk, and open it up with a hex editor. In the case of a 7z file you should see the byte sequence 37 7a bc af 27 1c somewhere. Most likely at the beginning of the file or at EOF-112 bytes. In the case of a gzip file, it should be 1f 8b at the beginning of the file.

Given that you have gzip in the content-encoding field: If you get a 7z file, you can be certain that requests has parsed content-encoding and properly decoded it for you. If you get a gzip file, it could mean two things. Either requests has not decoded anything, of the file is indeed a gzip file, as it could be a gzip file sent with the gzip encoding. Which would mean that it's doubly compressed. This would not make any sense, but, depending on the server this could still happen.

You could simply try to run gunzip on the console and see what you get.