Python中“收集并执行”的最佳实践

Question

I just fund myself implementing a timer-based version of "handle a list of events as a bunch" in order to safe resources - again - and I'm wondering whether there is a nice common pythonic approach.

You probably know this: you're handling recurring events like mouse movements, file system changes etc. and you have to do some calculation as a reaction to those events but it would be great if you could use a little break in the stream of events to handle them in a bunch. Maybe because older events get invalidated by newer events (and it's enough to handle the oldest ones) or because events can somehow be squashed together.

Examples are: mouse movements (draw only latest position), "auto save" in editors or auto-sync on file systems, or (in my example) monitoring file system changes and re-compile something.

Usually I look up how to use a Timer and think about how I could avoid an extra thread and come up with some semi-finished but complex solution for a - in my eyes - very simple problem. Lot of questions arise:

• how to avoid concurrent handling (eg if I use a threading.Timer and start a thread doing the work)
• how to make sure there is a time limit for events to be handled (in case of a continuous incoming of events without break)
• how to avoid threads if possible
• how to avoid creating an overly-complex framework
• (you name it)

What I'd like to have is something which works like this:

timer = SomeContinuousTimer()
new_events = []
while True:
   event = wait_for(inotify_adapter.event_gen(), timer.timeout())
   if event == timer.TIMEOUT:
       my_handler_func(new_events)
   else:
       new_events.append(event)
       timer.restart(1500)

But wait_for would have to act like select and for this I'd need file descriptors and the above code is already a bit more than I would actually expect it to be.

What I would be really glad about to have would be used like this:

bunch_handler = BunchHandler()
new_events = []

def read_events():
    for event in inotify_adapter.event_gen():
        new_events.append(event)

while True:
    # will run `read_events` asynchronously until 1.5sec have passed since the
    # last event
    bunch_handler.read(read_fn=read_events, bunch_wait=1500)

    handle_events(new_events)

Is this a typical scenario I should use async / await for? Are there frameworks for the case where async is not an option? Is there an async framework for this exact scenario?

Answer 1

This is not nice but it does what I want and might act as an example which shows, what I'm talking about:)

import asyncio
import time

async def event_collector(*, listener_fn, bunch_wait=1.0, max_wait=2.0):
    """Wait for (but don't handle) events and wait for a maximum of @bunch_wait seconds after the
    last event before returning. Force return after @max_wait seconds"""
    max_time_task = asyncio.Task(asyncio.sleep(max_wait))
    while True:
        resetable = asyncio.Task(asyncio.sleep(bunch_wait))
        done, _ = await asyncio.wait(
            {listener_fn.__anext__(), resetable, max_time_task},
            return_when=asyncio.FIRST_COMPLETED)
        if resetable in done or max_time_task in done:
            return
        resetable.cancel()


async def generate_events(events):
    """Simulates bursts of events with side-effects"""
    while True:
        for i in range(5):
            await asyncio.sleep(.01)
            events.append(i)
            print("*" * len(events))
            yield
        await asyncio.sleep(3.200)


def handle_events(events):
    """Simulates an event handler operating on a given structure"""
    print("Handle %d events" % len(events))
    events.clear()


async def main():
    new_events = []
    t = time.time()
    while True:
        await event_collector(listener_fn=generate_events(new_events), bunch_wait=1.1, max_wait=2.2)

        now = time.time()
        print("%.2f" % (now - t))
        t = now
        handle_events(new_events)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

This approach has some shortcomings: * you need to listen for events asynchronously using async * event_collector will return after max_wait seconds regardless whether any events have been seen yet (so it acts like a timeout if no events occur) * instead of resetting a timer, a new one gets created every time