Python多处理 - TypeError：出于安全原因，不允许选择AuthenticationString对象

Question

I'm having the following Problem. I want to implement a web crawler, so far this worked but it was so slow, that I tried to use multiprocessing for fetching the URLs. Unfortunately I'm not very experienced at this field. After some reading the easiest way seemed to me to use the map method from multiprocessing.pool for this.

But I constantly get the following error:

TypeError: Pickling an AuthenticationString object is disallowed for security reasons

I found very few cases with the same error and they unfortunately did not help me.

I created a stripped version of my code which can reproduce the error:

import multiprocessing

class TestCrawler:
    def __init__(self):
        self.m = multiprocessing.Manager()
        self.queue = self.m.Queue()
        for i in range(50):
            self.queue.put(str(i))
        self.pool = multiprocessing.Pool(6)



    def mainloop(self):
        self.process_next_url(self.queue)

        while True:
            self.pool.map(self.process_next_url, (self.queue,))                

    def process_next_url(self, queue):
        url = queue.get()
        print(url)


c = TestCrawler()
c.mainloop()

I would be very thankful about any help or suggestion!

Answer 1

Question : But I constantly get the following error:

The Error you'r getting is missleading, the reason are

self.queue = self.m.Queue()

Move the Queue instantiation Outside the class TestCrawler .
This leads to another Error:

NotImplementedError: pool objects cannot be passed between processes or pickled

The reason are:

self.pool = multiprocessing.Pool(6)

Both Errors are indicating that pickle can't find the class Members .

Note : Endless Loop!
Your following while Loop leads to a Endless Loop! This will overload your System!
Furthermore, your pool.map(... starts only one Process with one Task!

  while True: self.pool.map(self.process_next_url, (self.queue,)) 

I suggest reading The Examples that demonstrates the use of a pool

---

Change to the following:

class TestCrawler:
    def __init__(self, tasks):
        # Assign the Global task to class member
        self.queue = tasks
        for i in range(50):
            self.queue.put(str(i))

    def mainloop(self):
        # Instantiate the pool local
        pool = mp.Pool(6)
        for n in range(50):
            # .map requires a Parameter pass None
            pool.map(self.process_next_url, (None,))

    # None is passed
    def process_next_url(self, dummy):
        url = self.queue.get()
        print(url)

if __name__ == "__main__":
  # Create the Queue as Global
  tasks = mp.Manager().Queue()
  # Pass the Queue to your class TestCrawler
  c = TestCrawler(tasks)
  c.mainloop()

---

This Example starts 5 Processes each processing 10 Tasks(urls):

class TestCrawler2:
    def __init__(self, tasks):
        self.tasks = tasks

    def start(self):
        pool = mp.Pool(5)
        pool.map(self.process_url, self.tasks)

    def process_url(self, url):
        print('self.process_url({})'.format(url))

if __name__ == "__main__":
    tasks = ['url{}'.format(n) for n in range(50)]
    TestCrawler2(tasks).start()

Tested with Python: 3.4.2