如何将FileTransport干净地添加到Asyncio？

Question

I'm writing an application which reads text data and acts on it. The text data could come from a TCP port, or from a text file (which contains data earlier read from the TCP port and archived). I'm writing it in Python 3, and using asyncio seems like the obvious tool to use.

It's straightforward to use the Streams API open_connection() to open the TCP port and read from it. The asyncio architecture has the concept a Transport and a Protocol for the lower and upper layers of input-output. So, it seems I should implement a Transport for reading text from a file, and pass it to the Protocol. That will let me keep the rest of my application decoupled from whether the text data came from the TCP port or the file.

But I'm having a hard time figuring out how to tell asyncio to use my preferred Transport.

• the Streams API open_connection() has a parameter list which is all about the TCP port Transport, with no way to specify a different Transport, much less parameters like file path.
• open_connection() turns around and calls loop.create_connection() . This is just as specialised for the TCP Port Transport. Still now way to provide a different Transport.
• The implementation of loop.create_connection() gets its Transport object from either self._make_ssl_transport() or self._make_socket_transport() . These have alternative implementations in asyncio.selector_events.BaseSelectorEventLoop and asyncio.proactor_events.BaseProactorEventLoop , so we are clearly past the point where a File Transport ought to have been selected.

Am I missing some place where asyncio lets me tell it what Transport to use? Or is asyncio really coded down to its roots to use its own TCP port and UDP datagram Transports, and nothing else?

If I want to allow the possibility of using my own Transport with asyncio , it looks like I have to extend the event loop, or write more a flexible alternative create_connection() that is coded to a particular event loop implementation. That seems like a lot of work, and vulnerable to changes in the implementation.

Or, is it a foolish errand to handle file input with a Transport? Should I instead structure my code to say:

if (using_tcp_port): await asyncio.open_connection(....) else: completely_different_file_implementation(....)

Answer 1

According to the documentation of API create_connection() , it takes a protocol and creates a streaming transport, which is a TCP connection. So it is not supposed to be an API for custom transports.

However, the idea to reuse the same protocol for either TCP transports or custom file transports is valid. It won't be a "completely different implementation", but at least not using create_connection() . Let's assume it is read_file() :

def my_protocol_factory():
    return your_protocol

if using_tcp_port:
    transport, protocol = await loop.create_connection(my_protocol_factory, host, port)
else:
    transport, protocol = await read_file(loop, my_protocol_factory, path_to_file)

Then you would have something like this:

from asyncio import transports

import aiofiles  # https://github.com/Tinche/aiofiles


def read_file(loop, protocol_factory, path):
    protocol = protocol_factory()
    transport = FileTransport(path, loop)
    transport.set_protocol(protocol)
    return transport, protocol


class FileTransport(transports.ReadTransport):
    def __init__(self, path, loop):
        super().__init__()
        self._path = path
        self._loop = loop
        self._closing = False

    def is_closing(self):
        return self._closing

    def close(self):
        self._closing = True

    def set_protocol(self, protocol):
        self._protocol = protocol
        self._loop.create_task(self._do_read())

    def get_protocol(self):
        return self._protocol

    async def _do_read(self):
        try:
            async with aiofiles.open(self._path) as f:
                self._loop.call_soon(self._protocol.connection_made, self)
                async for line in f:
                    self._loop.call_soon(self._protocol.data_received, line)
                    if self._closing:
                        break
                self._loop.call_soon(self._protocol.eof_received)
        except Exception as ex:
            self._loop.call_soon(self._protocol.connection_lost, ex)
        else:
            self._loop.call_soon(self._protocol.connection_lost, None)