Client/server on python-twisted

Question

I am trying to make a Twisted application that is an XMLRPC server on the one hand and a client of special asymmetric binary protocol on the other (let's call it "binary") Now it works like this:

1. Receive xmlrpc request and return request id (I am server)
2. Make request to the "binary" server (I am client)
3. xmlrpc client makes polling requests with request id that was given in (1).
4. Poll request returns either "try again later" or real result, if it is already given by "binary" protocol.

Now it works, but I want to discard polling procedures.

But while I am inside xmlrpc request processing procedure, "binary" protocol exchange doesn't perform, so I never get a result from "binary" server.

What can you suggest? Separate threads? Something else?

from twisted.internet import reactor

reactor.connectTCP(globalconf.cfgBinServerAddr, globalconf.cfgBinServerPort, BinFactory(binProtocol))

reactor.listenTCP(globalconf.xmlrpcPort, server.Site(xmlrpcProtocol))
reactor.run()

Answer 1

1. Receive xmlrpc request. Compute request id if you need it to make request to the "binary" server. Do not return from the method
2. Make asynchronous request to the "binary" server (you are client) in the same method. Do not return yet
3. Drop this step (the xmlrpc request is not complete yet, nothing to poll)
4. Return a deferred "real result". You are done. xmlrpc machinery will return real result when it is ready to the xmlrpc client

Here's an example xmlrpc server that returns a deferred subprocess output:

#!/usr/bin/env python
from timeit import default_timer as timer
from twisted.internet import reactor, utils
from twisted.web import xmlrpc, server

class Example(xmlrpc.XMLRPC):
    def xmlrpc_echo(self, x):
        """Sanity check."""
        return x
    def xmlrpc_getoutput(self):
        shell_command = "echo before sleep; sleep 10; echo after sleep"
        start = timer()
        d = utils.getProcessOutput("/bin/sh", ["-c", shell_command])
        print("Deferred created in %.2f seconds" % (timer() - start,)) # instant
        return d

reactor.listenTCP(9657, server.Site(Example()))
reactor.run()

And the corresponding xmlrpc client:

#!/usr/bin/env python
from timeit import default_timer as timer
import xmlrpclib

s = xmlrpclib.Server('http://localhost:9657/')

def report_delay(func, *args):
    start = timer()
    print("Result %r took us %.2f seconds" % (func(*args), timer() - start))

report_delay(s.echo, "Mor-ee-air-teeeee") # should be instant
report_delay(s.getoutput) # should be ~10 seconds