Feasibility of using pipe for ruby-python communication

Question

Currently, I have two programs, one running on Ruby and the other in Python. I need to read a file in Ruby but I need first a library written in Python to parse the file. Currently, I use XMLRPC to have the two programs communicate. Porting the Python library to Ruby is out of question. However, I find and read that using XMLRPC has some performance overhead. Recently, I read that another solution for the Ruby-Python conundrum is the use of pipes. So I tried to experiment on that one. For example, I wrote this master script in ruby:

    (0..2).each do
      slave = IO.popen(['python','slave.py'],mode='r+')
      slave.write "master"
      slave.close_write
      line = slave.readline
      while line do
        sleep 1
        p eval line
        break if slave.eof
        line = slave.readline
      end
    end

The following is the Python slave:

    import sys

    cmd = sys.stdin.read()
    while cmd:
      x = cmd
      for i in range(0,5):
        print "{'%i'=>'%s'}" % (i, x)
        sys.stdout.flush()
        cmd = sys.stdin.read()

Everything seems to work fine:

    ~$ ruby master.rb
    {"0"=>"master"}
    {"1"=>"master"}
    {"2"=>"master"}
    {"3"=>"master"}
    {"4"=>"master"}
    {"0"=>"master"}
    {"1"=>"master"}
    {"2"=>"master"}
    {"3"=>"master"}
    {"4"=>"master"}
    {"0"=>"master"}
    {"1"=>"master"}
    {"2"=>"master"}
    {"3"=>"master"}
    {"4"=>"master"}

My question is, is it really feasible to implement the use of pipes for working with objects between Ruby and Python? One consideration is that there may be multiple instances of master.rb running. Will concurrency be an issue? Can pipes handle extensive operations and objects to be passed in between? If so, would it be a better alternative for RPC?

Answer 1

Yes. No. If you implement it, yes. Depends on what your application needs.

Basically if all you need is simple data passing pipes are fine, if you need to be constantly calling functions on objects in your remote process then you'll probably be better of using some form of existing RPC instead of reinventing the wheel. Whether that should be XMLRPC or something else is another matter.

Note that RPC will have to use some underlying IPC mechanism, which could well be pipes. but might also be sockets, message queues, shared memory, whatever.