Python生成器的多个客户端？

Question

As a follow up to this question I am trying to circumvent the list building exemplified by range(int(1e8)) using a generator xrange(int(1e8)) . Where the xrange is just an example for a process that produces a long sequence of values. (Please assume it can not be easily reproduced.) Some more background is, I have a long list of timestamp/value pairs that I want to do some processing on (sort of time-series). I try to avoid pulling these into memory as a whole, because that's a lot of data.

I thought it would be cool, if I could apply multiple processing units simultaneously to this stream of data produced by my generator. The first idea was to use itertools.tee() , eg:

from itertools import tee
g1,g2 = tee(xrange(int(1e8)),2)
sum(g1), sum(g2)

But then I found that only the first sum() would use the generator, while tee() internally builds a list again (Which I wanted to avoid.).

So I thought, I'm in need for a asynchronous solution, ie one that would allow each sum() do an update every generator step. The things that came in mind where

• coroutines
• multiprocessing
• twisted

But me having neither really used before, and partly I can not even tell whether the approaches might work, or be effective/efficient/performant.

From this point, I would gladly appreciate any suggestions from the audience!

---

Update

I wanted to avoid the callback based solution , as it apparantly decreases performance significantly (This is how it's currently implemented.). I have added some profiling below (please add comments if the test isn't objective):

class SinkA:
  def __init__(self, src):
    for i in src: pass

class SinkB:
  def f(self,i):
    pass

class Source:
  def __iter__(self):
    for i in xrange(int(1e4)):
      yield i

def t1():
  src = Source()
  snk = SinkA(src)

def t2():
  src = Source()
  snk = SinkB()
  for i in src: snk.f(i)

if __name__ == "__main__":
    from timeit import Timer
    n = 1000
    t = Timer("t1()", "from __main__ import t1, t2, SinkA, SinkB, Source")
    print "%.2f usec/pass" % (1000000 * t.timeit(number=n)/n) # 612.11 usec/pass
    t = Timer("t2()", "from __main__ import t1, t2, SinkA, SinkB, Source")
    print "%.2f usec/pass" % (1000000 * t.timeit(number=n)/n) # 1933.39 usec/pass

---

Update 2

What more can I say? I have this callback-based solution, that appears to be inefficient. The generator-based approach appears promising, but I have too little experience with that kind of programming, especially when it comes to more sophisticated things as coroutines, or the twisted library. To sum up, I have multiple consumers for a process that generates lots of data, and I have spotted some potential approaches. Now I'm looking for qualified statements by experienced users that probably have accomplished similar tasks before. Statements that address what approach could be appropriate, how the approaches relate to each other. Or what other approaches I might have missed after all.

Answer 1

As a generic approach, I would replace the generator's pull model with callbacks, and, probably, wrap the generator, like this:

def walk(gen, callbacks):
    for item in gen:
        for f in callbacks:
            f(item)

If your processors are in separate threads and you want them to block on waiting, you can register Queue.put (or anything equivalent) as a callback for each processor, and poll those queues independently. This will allow you to use both broadcasting and worker-pool models if you need so.

Edit

Another solution would be to use coroutines :

def source(self, *dests):
    for i in xrange(int(1e4)):
        for dest in dests:
            dest.send(i)

def sink():
    while True:
        i = yield

def t3():
    snk = sink()
    snk.next() # activate the coroutine
    source(snk)

if __name__ == '__main__':

    from timeit import Timer
    n = 1000
    t = Timer("t3()", "from __main__ import source, sink, t3")
    print "%.2f usec/pass" % (1000000 * t.timeit(number=n)/n) # 872.99 usec/pass

Looks fast enough. Basically, coroutines are the inverted generators, you pull from generator, push to coroutine.

Answer 2

You don't really address this, but do you want each consumer to see the exact same data (in which case tee is probably the best solution), or not?

If not, then you can simply have each consumer read from the one generator object.

If you do want them to get the exact same data, try tee (uses more memory) vs two generators (more IO), and see which is faster.

As to your timings, what your data show is simply that there is an overhead to multiple function calls, and that one of your methods avoids intermediate function calls.

If you want to improve performance, try running this on PyPy, which has a hotspot-optimising JIT.

Answer 3

Since generators are cheap in memory, why don't you simply use two independent generators?

g1 = xrange(int(1e8))
g2 = xrange(int(1e8))
sum(g1), sum(g2)

Answer 4

Solution for sharing python generators with tests:

https://gist.github.com/earonesty/cafa4626a2def6766acf5098331157b3

Example of use:

def mygen():
       yield from [1,2,3]

m1 = Muxer(mygen)
m2 = Muxer(mygen)

consume1(m1)
consume2(m2)

Code for muxer.py :

import queue
from threading import Lock
from collections import namedtuple

class Muxer():
    Entry = namedtuple('Entry', 'genref listeners, lock')

    already = {}
    top_lock = Lock()

    def __init__(self, func, restart=False):
        self.restart = restart
        self.func = func
        self.queue = queue.Queue()

        with self.top_lock:
            if func not in self.already:
                self.already[func] = self.Entry([func()], [], Lock())
            ent = self.already[func]

        self.genref = ent.genref
        self.lock = ent.lock
        self.listeners = ent.listeners

        self.listeners.append(self)

    def __iter__(self):
        return self

    def __next__(self):
        try:
            e = self.queue.get_nowait()
        except queue.Empty:
            with self.lock:
                try:
                    e = self.queue.get_nowait()
                except queue.Empty:
                    try:
                        e = next(self.genref[0])
                        for other in self.listeners:
                            if not other is self:
                                other.queue.put(e)
                    except StopIteration:
                        if self.restart:
                            self.genref[0] = self.func()
                        raise
        return e

    def __del__(self):
        with self.top_lock:
            try:
                self.listeners.remove(self)
            except ValueError:
                pass
            if not self.listeners and self.func in self.already:
                del self.already[self.func]

Answer 5

我建议你查看如何使用协同程序实现这一点，更具体地说是这个广播示例