Erlang: port to Python instance not responding

Question

I am trying to communicate to an external python process through an Erlang port. First, a port is opened, then a message is sent to the external process via stdin. I am expecting a corresponding reply on the process's stdout.

My attempt looks like this:

% open a port
Port = open_port( {spawn, "python -u -"},
                  [exit_status, stderr_to_stdout, {line, 1000000}] ).

% send a command to the port
true = port_command( Port, "print( \"Hello world.\" )\n" ).

% gather response
% PROBLEM: no matter how long I wait flushing will return nothing
flush().

% close port
true = port_close( Port ).

% still nothing
flush().

I realize that someone else on Stackoverflow tried to do something similar but the proposed solution apparently doesn't work for me.

Also, I see that a related post on Erlang Central is starting a Python script through an Erlang port but it is not the Python shell itself that is invoked.

I have taken notice of ErlPort but I have a whole script to be executed in Python. If possible, I wouldn't want to break up the script into single Python calls.

Funny enough, doing it with bash is no problem:

Port = open_port( {spawn, "bash"},
                  [exit_status, stderr_to_stdout, {line, 1000000}] ).

true = port_command( Port, "echo \"Hello world.\"\n" ).

So the above example gives me a "Hello world." on flushing:

3> flush().
Shell got {#Port<0.544>,{data,{eol,"Hello world."}}}
ok

Just what I wanted to see.

• Ubuntu 15.04 64 bit
• Erlang 18.1
• Python 2.7.9

Edit: I have finally decided to write a script file (with a shebang) to disk and execute the script file instead of piping the script to the language interpreter for some languages (like Python).

I suspect, the problem has to do with the way some interpreters buffer IO, which I just can't work around, making necessary this extra round to disk.

Answer 1

As you've discovered, ports don't do what you'd like for this problem, which is why alternatives like ErlPort exist. An old workaround for this problem is to use netcat to pipe commands into python so that a proper EOF occurs. Here's an example session:

1> PortOpts = [exit_status, stderr_to_stdout, {line,1000000}].
[exit_status,stderr_to_stdout,{line,1000000},use_stdio]
2> Port = open_port({spawn, "nc -l 51234 | python"}, PortOpts).
#Port<0.564>
3> {ok, S} = gen_tcp:connect("localhost", 51234, []).
{ok,#Port<0.565>}
4> gen_tcp:send(S, "print 'hello'\nprint 'hello again'\n").
ok
5> gen_tcp:send(S, "print 'hello, one more time'\n").
ok
6> gen_tcp:close(S).
ok
7> flush().
Shell got {#Port<0.564>,{data,{eol,"hello"}}}
Shell got {#Port<0.564>,{data,{eol,"hello again"}}}
Shell got {#Port<0.564>,{data,{eol,"hello, one more time"}}}
Shell got {#Port<0.564>,{exit_status,0}}
ok

This approach opens a port running netcat as a listener on port 51234 — you can choose whatever port you wish to, of course, as long as its not already in use — with its output piped into python . We then connect to netcat over the local TCP loopback and send python command strings into it, which it then forwards through its pipe to python. Closing the socket causes netcat to exit, which results in an EOF on python's stdin, which in turn causes it to execute the commands we sent it. Flushing the Erlang shell message queue shows we got the results we expected from python via the Erlang port.