Python - 通过套接字与子进程通信

Question

I am looking to replicate the way programs like Git and Rsync communicate and transfer data over an SSH connection, but in Python. I understand that these programs fork and exec an SSH command that starts a process on the server side and communication is achieved by the parent processing talking to the STDIN and STDOUT of the forked child process.

In CI have seen this done by creating a Unix socket pair (s0, s1), forking the process, pointing the stdin/stdout of the forked process to s1 on the child process, and then reading and writing to the socket s0 on the parent process.

I want to do the same thing in Python3. As a proof of concept, here is my implementation of a toy remote shell that sends commands down a socket and receives the output from the same socket:

import subprocess
import socket


ssh_cmd = 'ssh -q -T ubuntu@xxxxxxxx'
s_local, s_remote = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)

ssh = subprocess.Popen(ssh_cmd, shell=True,
                       stdout=s_remote,
                       stdin=s_remote,
                       close_fds=True)
s_remote.close()

s_local.send('ls -l\n'.encode())
print(s_local.recv(1024).decode())
s_local.send('uname -a\n'.encode())
print(s_local.recv(1024).decode())
s_local.close()
ssh.kill()

This kind of works. But I can deadlock it if I send 'true \\n' down the socket because 'recv' is blocking so that's not great.

1. Is this a sensible/common thing to do in network programming?
2. What is a more idiomatic way of doing this using the Python3 standard libraries that doesn't deadlock?

Answer 1

I connected a web browser to the stdin and stdout of a command running on the remote end of a SSH connection. It sounds like you're trying to do something similar: connect a socket to a program running on a remote machine so that clients can connect to a host/port and use said remote program.

The first thing you need to do is establish a SSH connection. I use an external ssh program (either ssh or plink depending on OS) in a subprocess. You could just use class subprocess.Popen to do that. The subprocess launches the (external) ssh application to connect to the remote host and runs the required command. That then sits there listening on its 'stdin' and willing to reply on its 'stdout'.

(You could possibly use Python SSH implementation such as Paramiko here)

At a very high level (where remote_command is the thing to run on the far end):

import subprocess
command = ['ssh', 'user@host', 'remote_command']

sproc = subprocess.Popen(command, shell=False,
                         stdin = subprocess.PIPE,
                         stdout = subprocess.PIPE,
                         stderr = subprocess.PIPE)

This leaves you with a subprocess that you can interact with through its stdin , stdout and stderr .

Get this bit working first.

The next thing you need is to bind a socket, listen and accept a connection. Then read from the client and write to the server. I wrote a lightweight Socat (inspired by Socat ) class:

import socket

class Socat(object):

EOT = '\004\r\n'

def __init__(self, binding, stream, eot = EOT):

  self.stream = stream
  self.eot = eot

  self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  self.socket.bind(binding)
  self.socket.listen(5)

  while True:
    print "waiting for connection"
    (clientsocket, address) = self.socket.accept()
    print "connection from %s:%s" % address
    data = clientsocket.recv(1024)
    self.stream.write(data)
    while data:
      data = self.stream.readline()
      if data:
        print "read %s bytes" % len(data)
        if data == self.eot:
          print "read termination signature"
          clientsocket.close()
          data = None
        else:
          clientsocket.send(data)

You should read up on Sockets: this and this are useful here.

You can then use this to connect to your remote_command :

Socat( ('localhost', 8080), sproc)

There are some missing bits

You'll notice the socat class takes in a stream (unlike my example) that has write and readline methods. I actually encapsulated the SSH connection in a class but I'll leave that as an exercise for the reader!

(Basically a class Stream that contains the sproc and provides write and readline methods - effectively sproc.stdin.write() and sproc.stdout.readline() - you get the idea!)

Also, you need a protocol to allow remote_command to signal the end of a transmission. I used the ASCII EOT character and my remote program sends that at the end of a response. The socat uses that to close the client connection and accept a new one. Depending on your use-case you will need to come up with a protocol that works for you.

Finally

All of the above can be improved upon. What's above is my first cut because it is simple to see the inner workings. I have since extended it to run the accept loop in a thread and gracefully handle disconnects and so-on.

But hopefully what's above is a reasonable starting point.