i use multiprocessing.managers.BaseManager to manager a Queue at server side,i try to use this queue in another python script which uses a process pool,but i always got the error message as follow, and the main code of this python script is as follow too, and it will start with the run() method.
File "/usr/lib/python2.7/multiprocessing/connection.py", line 435, in answer_challenge
raise AuthenticationError('digest sent was rejected')
AuthenticationError: (AuthenticationError('digest sent was rejected',), <function RebuildProxy at 0x7ff8de0b8320>, (<function AutoProxy at 0x7ff8de0b7938>, Token(typeid='Queue', address=('localhost', 12345), id='7f4624039cd0'), 'pickle', {'exposed': ('cancel_join_thread', 'close', 'empty', 'full', 'get', 'get_nowait', 'join_thread', 'put', 'put_nowait', 'qsize')}))
def __init__(self, spider_count=cpu_count()):
self._spider_count = spider_count
mgr = MyManager(address=('localhost', 12345), authkey='xxxxx')
server = mgr.connect()
self._queue = mgr.Queue()
def run(self):
pool = Pool(self._spider_count)
while not self._queue.empty():
#add some control on q.get() if queue is empty
pool.apply_async(self.startCrawl, (self._queue.get(),))
pool.close()
pool.join()
but when i use it at a single thread, it works well, when use the pool, this error message raised.
This sounds like the issue described in http://bugs.python.org/issue7503
In practice all processes using the manager should have
current_process().authkey
set to the same value.
The fix, then, would be to assign in __init__
:
multiprocessing.current_process().authkey = 'xxxxx'
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.