Python ftplib：上传到ftp的bz2文件有时会损坏

Question

I am using ftplib module to upload many bz2 files to a ftp (about 1000 files per day, and every file is about 5 MB, storing a numpy array). Once in a while, some uploaded files are corrupted. When I try to read it using bz2 and numpy, I get the error "IOError: invalid data stream". If I try to uncompress it using software such as WinRAR, I get the message "Checksum error in filename . The file is corrupted."

The code for uploading the data has nothing special. Basically it looks like this:

while True:
    try:
        fidFile = open(fileName, 'rb')
        ftp.storbinary('STOR '+fileName, fidFile)
        fidFile.close()
        break
    except:
        continue

For the corrupted files, if I upload them again using the same code, most of the time I can get the good files.

Using a different ftp does not eliminate this problem.

I also noticed that the corrupted file has exactly the same bytes as the good file. I guess all necessary information has been uploaded, so I really do not understand why the file is corrupted.

Answer 1

Workaround for this problem can be:

def upload(fileName):
    try:
        fidFile = open(fileName, 'rb')
        ftp.storbinary('STOR '+fileName, fidFile)
        fidFile.close()
    except Exception as e:
        print(e)
        upload(fileName)
        # Can be improved by restricting retry limit.

It keeps trying to upload the file and will go off ones it is uploaded.