python 3.2 pickle.load随机导致EOFError

Question

I encounter a very strange error. I have a cronjob that run daily: it opens a pickled file, load that file and then process the data. At the end of the script, it will save the data to that file for next day processing.

Most of the time, this script runs fine but there's some time (I would say twice a month) when opening that file, an EOFError is raised:

Traceback (most recent call last):
  File "read.py", line 11, in <module>
    SellerDictionarycf=pickle.load(SellerDict)
EOFError

I strongly believe that there's no other process is working with that file and I'm 100% sure that I write the data, close that file before reading it.

My code snipet to read the file at beginning of the script:

SellerDict=open('/home/hostadl/SellerDictab','rb')
SellerDictionaryab=pickle.load(SellerDict)
SellerDict.close()

My code snipet to write the file at the end of the script:

SellerDict=open('/home/hostadl/SellerDictab','wb')
pickle.dump(SellerDictionaryab,SellerDict)
SellerDict.flush()
SellerDict.close()

I saved a copy of the corrupt file (file that raise EOFError when reading) and then check it with pickletools and here is the error:

python3.2 -m pickletools -o test.txt SellerDictab
Traceback (most recent call last):
  File "/usr/local/lib/python3.2/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/lib/python3.2/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.2/pickletools.py", line 2403, in <module>
    args.indentlevel, annotate)
  File "/usr/local/lib/python3.2/pickletools.py", line 1963, in dis
    for opcode, arg, pos in genops(pickle):
  File "/usr/local/lib/python3.2/pickletools.py", line 1874, in genops
    arg = opcode.arg.reader(pickle)
  File "/usr/local/lib/python3.2/pickletools.py", line 474, in read_unicodestring4
    "remain" % (n, len(data)))
ValueError: expected 2254 bytes in a unicodestring4, but only 0 remain

The test.txt still has data and here is the head and tail:

head test.txt
    0: \x80 PROTO      3
    2: }    EMPTY_DICT
    3: q    BINPUT     0
    5: (    MARK
    6: X        BINUNICODE 'bradey4357604769'
   27: q        BINPUT     1
   29: ]        EMPTY_LIST
   30: q        BINPUT     2
   32: ]        EMPTY_LIST
   33: q        BINPUT     3

tail test.txt
18933166: e                APPENDS    (MARK at 18930621)
18933167: ]            EMPTY_LIST
18933168: r            LONG_BINPUT 174342
18933173: (            MARK
18933174: X                BINUNICODE 'HP PhotoSmart C4480 All-in-One Printer'
18933217: r                LONG_BINPUT 174343
18933222: G                BINFLOAT   45.0
18933231: G                BINFLOAT   84104.0
18933240: X                BINUNICODE 'Salt Lake City,  UT '
18933265: r                LONG_BINPUT 174344

For the good file (with no EOFError) here is the head and tail:

head testgood.txt
    0: \x80 PROTO      3
    2: }    EMPTY_DICT
    3: q    BINPUT     0
    5: (    MARK
    6: X        BINUNICODE 'bj8016541577'
   23: q        BINPUT     1
   25: ]        EMPTY_LIST
   26: q        BINPUT     2
   28: (        MARK
   29: ]            EMPTY_LIST

tail testgood.txt
16569368: X                BINUNICODE 'Bought this beautiful dinner set but never opened it, I would love to get rod of it pleas tex with an offer...'
16569483: r                LONG_BINPUT 161843
16569488: J                BININT     20177894
16569493: X                BINUNICODE 'antonio8016497082'
16569515: r                LONG_BINPUT 161844
16569520: e                APPENDS    (MARK at 16569283)
16569521: e            APPENDS    (MARK at 16568508)
16569522: u        SETITEMS   (MARK at 16088830)
16569523: .    STOP

So what could be the cause of this problem (like strange characters in the string data)? I have no clue on this problem.

Answer 1

Looks like you ran out of disk space (or were unable to write the complete file for some similar reason).

A more idiomatic version (where you wouldn't need to worry about calling close/flush) would be:

with open('/home/hostadl/SellerDictab','rb') as SellerDict:
    SellerDictionaryab=pickle.load(SellerDict)

with open('/home/hostadl/SellerDictab','wb') as SellerDict:
    pickle.dump(SellerDictionaryab,SellerDict)