使用lxml.etree.iterparse的线程问题

Question

I have a thread which spawns multiple consumer processes that do some heavy processing work on large xml files.

My design for this was to use a simple single thread for parsing the inbound stream on the fly and shove new objects into a multiprocessing.queues.queue class contained within a buffer process manager. The process manager periodically checks the size of the queue and if consumption is letting the queue fill up too quickly it kicks off another consumer.

My problem is that the code to join to the closed queue when stream parsing has completed is executing before the xml has finished being parsed!? This doesn't seem to me to be how the following code is supposed to work. Keep in mind that the following code is completely single-threaded. It is neither called nor used by any SMP code:

clear_ok = False
context = lxml.etree.iterparse(response, events=('end',))
for event, elem in context:
    # Use QName to avoid specifying or stripping the namespace, which we don't need
    if lxml.etree.QName(elem.tag).localname.upper() in obj_elem_map:
        import_buffer.add(obj_elem_map[lxml.etree.QName(elem.tag).localname.upper()](elem=elem))
        clear_ok = True
    if clear_ok:
        elem.clear() #don't fill up a dom we don't need.
        clear_ok = False
results = import_buffer.finish() if block else import_buffer

when import_buffer.finish() is called the following happens:

def finish(self):
    '''
    Notifies the buffer that we are done filling it.
    This command binds to any processes still running and lets them
    finish and then copies and flushes the managed results list.
    '''
    # close the queue and wait until it is consumed
    self.queue.close()
    self.queue.join_thread()
    # make sure the consumers are done consuming the queue
    for csmr in self.running:
        csmr.join()
    # turn this into a list instead of a managed list
    result = list(self.results_list)
    del self.results_list[:]
    if self.callback:
        return self.callback(result)
    else:
        return result

However I'm getting an exception that that close() has been called on the queue before I've finished parsing?

Traceback (most recent call last):
  File "./tests/test_smp_framework.py", line 103, in test_kqb_parser_fromfile
    qkbobs = actions.queryQKB(file=fname)
  File "/Users/skyleach/src/smpparser/smpparser/api_actions.py", line 339, in queryQKB
    result = self.parseResponse(source=sourcefile)File "/Users/skyleach/src/smpparser/smpparser/smpapi.py", line 535, in parseResponse
    import_buffer.add(obj_elem_map[lxml.etree.QName(elem.tag).localname.upper()](elem=elem))
  File "/Users/skyleach/src/smpparser/smpparser/smpapi.py", line 212, in add
    self.queue.put(item)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/queues.py", line 81, in put
    assert not self._closed
AssertionError

Answer 1

This wasn't lxml's fault or multiprocessing's fault, it was a context assignment initialization problem. Basically I coded too fast and made a dumb mistake.

In my buffer class definition I was setting up the queue rather than in the init function. This means that all queues for all instances of the class were one queue defined when the module was imported by the import thread.

That's what I get for writing code too fast.