[英]pickle.dump meet RuntimeError: maximum recursion depth exceeded in cmp
I have noticed that it may be caused by beautifulsoup or recursive data structure. 我注意到它可能是由beautifulsoup或递归数据结构引起的。 however, the data structure that cause error seems no problem:
但是,导致错误的数据结构似乎没有问题:
class Movie:
def __init__(self, name="", dscore=0, mscore=0, durl="", murl=""):
self.name = name
self.dscore = float(dscore)
self.mscore = float(mscore)
self.durl = durl
self.murl = murl
def __str__(self):
return unicode(self.name) + u' / ' + unicode(self.dscore) + u' / ' + unicode(self.mscore) \
+ u' / ' + unicode(self.durl) + u' / ' + unicode(self.murl)
The statement causing the problem is: 导致问题的陈述是:
DataDict['MovieInfo'] = MovieInfo
and 和
pickle.dump(DataDict, f, True)
following is the function: 以下是功能:
def SaveData():
global LinkUrlQueue
global MovieSet
global MovieInfo
global LinkUrlSet
global MovieUrlQueue
DataDict = {}
DataDict['LinkUrlSet'] = LinkUrlSet
DataDict['MovieSet'] = MovieSet
#DataDict['MovieInfo'] = MovieInfo
DataDict['LinkUrlQueue'] = LinkUrlQueue
DataDict['MovieUrlQueue'] = MovieUrlQueue
f = open('MovieInfo.txt', 'wb')
for item in MovieInfo:
f.write(item.__str__().encode('utf8') + '\n'.encode('utf8'))
f.close()
try:
print 'saving data...'
f = open('spider.dat', 'wb')
pickle.dump(DataDict, f, True)
f.close()
except IOError as e:
print 'IOError, error no: %d' % e.no
print 'saved to spider2.dat'
pickle.dump(DataDict, open('spider2.dat', 'wb'))
time.sleep(10)
my complete source code: 我的完整源代码:
spider.py: http://paste.ubuntu.com/7149731/ spider.py: http://paste.ubuntu.com/7149731/
fetch.py: http://paste.ubuntu.com/7149732/ fetch.py: http://paste.ubuntu.com/7149732/
You can just download and run. 你可以下载并运行。
Besides, welcome any coding style suggestions 此外,欢迎任何编码风格的建议
Well... I finally solve the problem by myself... 嗯......我终于自己解决了这个问题......
The reason for this problem is that pickle cannot handle BEAUTIFULSOUP!!! 造成这个问题的原因是泡菜无法处理BEAUTIFULSOUP! Generally, it cannot handle html parser.
通常,它无法处理html解析器。
I realize that when passing arguments into my functions, I should convert them into str() or unicode() then do assignments, instead of remaining them as beautifulsoup objects... 我意识到,当将参数传递给我的函数时,我应该将它们转换为str()或unicode()然后进行赋值,而不是将它们保留为beautifulsoup对象......
thanks for everyone~ 谢谢大家〜
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.