Piping output from a python script to a remote machine?

Question

I have a python script that output/prints a large array (50MB in size) to the screen. This script is running on machineA . Is it at all possible to 'pipe' this output to machineB , where it will be processed?

I know I can save the output in a file on machineA and then send the file to machineB . I was just wondering if it is possible to do so without having to save the data first to a file (on machineA ).

Any solutions (python, bash shell, other ideas) will be appreciated.

Answer 1

This is a work for nc (netcat)... on machineB you do

nc -l 12345 | processing_program

This command will start netcat in "listen mode" waiting for a connection. Once the connection is established what is coming from the network will be sent to stdout and what is sent to stdin will be sent back to whoever connected.

After that you go on machineA and run

generating_program | nc machineB 12345

this will instruct netcat to start a connection to machineB (port 12345) and send whatever it gets from stdin over the wire. Whatever comes back from that connection is sent to standard output.

Answer 2

使用Netcat工具，您可以轻松地将数据从一台计算机传输到另一台计算机。

Answer 3

You can do it on different levels - here are a few options

1. use ssh to pipe myprog | ssh remotemachine myotherprog
2. use nfs (if going to a file
3. use netcat (nc)
4. use something like thrift

It depends on how solid & permanent the solution needs to be

Answer 4

Maybe something along the lines of

myPythonScript | ssh user@machine "sh -c 'cat >myfile.txt'"

WARNING: pulled entirely out of my head and untested!

Answer 5

On machineA:

python script-on-machineA.py --options | ssh machineB tee /file/on/machine/B.txt

Alternatively, on machineB:

ssh machineA python /path/to/script-on-machineA.py --options | cat > /file/on/machine/B.txt