EC2 Python泡菜失败

Question

I'm doing a calculation on EC2 using python and when I try to dump a pickle file containing a dictionary, the program just exists, no error notice or anything. The file is large, around 1 gig, but everything works fine on my laptop, just not on EC2. I'm using an m3.large instance with an attached EBS volume with plenty of space. In the following code snippet, it prints out "dumping now" and then.....nothing, no "dumping complete". No error is caught by 'except'. When I try to load the pickle file I get an EOF Error.

Thanks for any advice!

try:
    fp = open(pickleFile,"wb")
    print 'dumping now'
    pickle.dump(dataDict, fp)
    print 'dumping complete'
    fp.close()
except:
    fp = open('/Users/thisUser/Data/report.txt','w')
    fp.write('error writing pickle file')
    fp.close()

Answer 1

1. Especially if you're debugging file IO, try not to have your debugging depend on said file IO. If there are any permissions or disk issues, you'll never know since the error message won't be able to write either.

2. Try running watch "ls -lh /your/file" , then running your script, to see if the file writing stops suddenly, or is just continuing at a slow pace.

3. Try Ctrl-C if your script is hung and your watch looks static, and check out where in the traceback your function is spending its time.

4. Try pickling elements/keys of the dictionary individually--the pickling may be crashing on an individual complex element.

5. Try cPickle

6. Compare your versions at home and on ec2: import cPickle; print cPickle.__version__ import cPickle; print cPickle.__version__

7. Try different pickling versions.

I'm not sure which of these might help, but one of them might.