使用 Python 压缩大文件，速度非常快

Question

I'm using this function to gzip a file:

---

def zip_file(path_data,path_zip,File):
    with open(os.path.join(path_data,File), "rb") as f_in, gzip.open(os.path.join(path_zip,File) + ".gz", "wb") as f_out:
        shutil.copyfileobj(f_in, f_out,length=16*1024*1024)

---

But it takes 1604.954 seconds to gzip a 14 GB file with 4 columns, I have to process 96 files like this.

Answer 1

Add a parameter to your gzip.open with compresslevel=1 . You can play with the level between 1 and 5 (default is 6, which apparently you don't like). See where you prefer the trade off in time vs. compression ratio.

By the way, you shouldn't call it "zip_file". It is not a zip file, which is an entirely different thing from a gzip file. Call it "gzip_file", or something else.