Trying to get a Python client talk to a Java Server using thrift's TFileTransport and TFileProcessor

Question

I am trying to wrap a Java class to be called from Python using thrift's TFileTransport. I tried using two protocols TJSON and TBinary but I keep getting this exception

org.apache.thrift.transport.TTransportException: FileTransport error: bad event size
    at org.apache.thrift.transport.TFileTransport.readEvent(TFileTransport.java:327)
    at org.apache.thrift.transport.TFileTransport.read(TFileTransport.java:468)
    at org.apache.thrift.transport.TFileTransport.readAll(TFileTransport.java:439)
    at org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
    at org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
    at org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
    at org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.thrift.transport.TFileProcessor.processUntil(TFileProcessor.java:69)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:102)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:111)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:118)
    at com.netflix.suro.client.SendToPyServer.startThriftServer(SendToPyServer.java:51)
    at com.netflix.suro.client.SendToPyServer.main(SendToPyServer.java:67)

This is how my Python client looks:

 def __init__(self):
        self.outFile=open("../../ThriftFile.in","a")
        self.transport = TTransport.TFileObjectTransport(self.outFile)
        self.protocol = TJSONProtocol.TJSONProtocol(self.transport)
        self.client = sendPyInterface.Client(self.protocol)
        self.transport.open()

    def send(self,routingKey, message):
        self.transport.write(pickle.dumps(self.client.send_send(routingKey, message))) 


    def configClient(self,configurationDict):       
        self.transport.write(pickle.dumps(self.client.send_ClientConfig(configurationDict)))    

if __name__ == "__main__":
    SuroClient=SuroPyClient()
    configurationDict={"ClientConfig.LB_TYPE":"static","ClientConfig.LB_SERVER":"localhost:7101"}
    SuroClient.configClient(configurationDict)
    SuroClient.send("routingKey", "testMessage")

and this is my server:

public static void startThriftServer(SendPyInterface.Processor processor) {
            try {
                File input = new File("src/main/java/com/netflix/suro/client/ThriftFile.in");
                if(!input.exists()){
                    input.createNewFile();
                }

                File output = new File("src/main/java/com/netflix/suro/client/ThriftFile.out");
                if(!output.exists()){
                    output.createNewFile();
                }

                TFileTransport inputFileTransport = new TFileTransport(input.getAbsolutePath(), true);
                TFileTransport outputFileTransport = new TFileTransport(output.getAbsolutePath(), false);
                System.out.println(input.getAbsolutePath());
                System.out.println(input.length());

                inputFileTransport.open();
                outputFileTransport.open();
                System.out.println(inputFileTransport.getBytesRemainingInBuffer());
                inputFileTransport.setTailPolicy(tailPolicy.WAIT_FOREVER);
                System.out.println("Wait ...");
                System.out.println(inputFileTransport.getBuffer());
                TFileProcessor fProcessor = 
                        new TFileProcessor(processor, new TJSONProtocol.Factory(), inputFileTransport, outputFileTransport);
                try {
                    fProcessor.processChunk();
                } catch (TTransportException e) {
                    e.printStackTrace();
                }


                System.out.println("File Thrift service started ...");
            } catch (Exception e) {
                e.printStackTrace();
            }

I have followed the example from this thread: Example on how to use TFileTransport in Thrift (Client/Server)

and from this post: http://theprogrammersguideto.com/thrift/blog/code/chapter-3-moving-bytes-with-transports/

Answer 1

According to the docs in the old Thrift wiki , the Java and C++ TFileTransport is actually a framed transport , which writes an additional 4-byte header preceding the data. Seems the names are a bit misleading here.

In contrast, the Python TFileObjectTransport used is just a simple wrapper. This means, that the transports are not compatible, hence the error message.

So instead of TFileTransport , the TSimpleFileTransport should be used with Java.