Non-blocking file access with Twisted

Question

I'm trying to figure out if there is a defacto pattern for file access using twisted. Lots of examples I've looked at (twisted.python.log, twisted.persisted.dirdbm, twisted.web.static) actually don't seem to worry about blocking for file access.

It seems like there should be some obvious interface, probably inheriting from abstract.FileDescriptor, that all file access should be going through it as a producer/consumer.

Have I missed something or is it just that the primary use for twisted in asynchronous programming is for networking and it hasn't really been worked out for other file descriptor operations, not worrying about the purity of non-blocking IO ?

Answer 1

I think you are looking for the fdesc module . For more information on non-blocking I/O in Python, you can also watch this video .

Answer 2

在Twisted中有一张打开的门票 - ＃3983 。

Answer 3

After plenty of searching, trial, and error, I finally figured how to use fdesc .

from __future__ import print_function

from twisted.internet.task import react
from twisted.internet import stdio, protocol
from twisted.internet.defer import Deferred
from twisted.internet.fdesc import readFromFD, setNonBlocking


class FileReader(protocol.Protocol):
    def __init__(self, filename):
        self.f = open(filename, 'rb')

    def dataReceived(self, data):
        self.transport.write(data)

    def connectionMade(self):
        fd = self.f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, self.dataReceived)

    def connectionLost(self, reason):
        self.f.close()

def main(reactor, filename):
    stdio.StandardIO(FileReader(filename))

[Edit: I also just figured out a simpler way that doesn't require the use of a protocol]

def getFile(filename):
    with open(filename) as f:
        d = Deferred()
        fd = f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, d.callback)
        return d


def main(reactor, filename):
    d = getFile(filename)
    return d.addCallback(print)

Run either like so:

react(main, ['/path/to/file'])

Answer 4

The fdesc module might be useful for asynchronously talking to a socket or pipe, but when given an fd that refers to an ordinary filesystem file, it does blocking io (and via a rather odd interface at that). For disk io, fdesc is effectively snake oil; don't use it.

As of May 2017, the only reasonable way to get async disk io in twisted is by wrapping synchronous io calls in a deferToThread .

Answer 5

I'm not sure what you want to achieve. When you do logging, then Python will make sure (by the global interpreter log) that log messages from several threads go into the file one after the other.

If you're concerned about blocking IO, then the OS adds default buffers for your files (usually 4KB), and you can pass a buffer size in the open() call.

If you're concerned about something else, then please clarify your question.