Using ssh and sed within a python script with os.system properly

Question

I am trying to run an ssh command within a python script using os.system to add a 0 at the end of a fully matched string in a remote server using ssh and sed .

I have a file called nodelist in a remote server that's a list that looks like this.

test-node-1
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21

I want to use sed to make the following modification, I want to search test-node-1 , and when a full match is found I want to add a 0 at the end, the file must end up looking like this.

test-node-1 0
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21

However, when I run the first command,

hostname = 'test-node-1'
function = 'nodelist'

os.system(f"ssh -i ~/.ssh/my-ssh-key username@serverlocation \"sed -i '/{hostname}/s/$/ 0/' ~/{function}.txt\"")

The result becomes like this,

test-node-1 0
test-node-2
...
test-node-11 0
test-node-12 0
test-node-13 0
...
test-node-21

I tried adding a \\b to the command like this,

os.system(f"ssh -i ~/.ssh/my-ssh-key username@serverlocation \"sed -i '/\b{hostname}\b/s/$/ 0/' ~/{function}.txt\"")

The command doesn't work at all.

I have to manually type in the node name instead of using a variable like so,

os.system(f"ssh -i ~/.ssh/my-ssh-key username@serverlocation \"sed -i '/\btest-node-1\b/s/$/ 0/' ~/{function}.txt\"")

to make my command work.

What's wrong with my command, why can't I do what I want it to do?

Answer 1

This code has serious security problems; fixing them requires reengineering it from scratch. Let's do that here:

#!/usr/bin/env python3
import os.path
import shlex  # note, quote is only here in Python 3.x; in 2.x it was in the pipes module
import subprocess
import sys

# can set these from a loop if you choose, of course
username = "whoever"
serverlocation = "whereever"
hostname = 'test-node-1'
function = 'somename'

desired_cmd = ['sed', '-i',
               f'/\\b{hostname}\\b/s/$/ 0/',
               f'{function}.txt']
desired_cmd_str = ' '.join(shlex.quote(word) for word in desired_cmd)
print(f"Remote command: {desired_cmd_str}", file=sys.stderr)

# could just pass the below direct to subprocess.run, but let's log what we're doing:
ssh_cmd = ['ssh', '-i', os.path.expanduser('~/.ssh/my-ssh-key'),
           f"{username}@{serverlocation}", desired_cmd_str]
ssh_cmd_str = ' '.join(shlex.quote(word) for word in ssh_cmd)
print(f"Local command: {ssh_cmd_str}", file=sys.stderr)  # log equivalent shell command
subprocess.run(ssh_cmd) # but locally, run without a shell

If you run this (except for the subprocess.run at the end, which would require a real SSH key, hostname, etc), output looks like:

Remote command: sed -i '/\btest-node-1\b/s/$/ 0/' somename.txt
Local command: ssh -i /home/yourname/.ssh/my-ssh-key whoever@whereever 'sed -i '"'"'/\btest-node-1\b/s/$/ 0/'"'"' somename.txt'

That's correct/desired output; the funny '"'"' idiom is how one safely injects a literal single quote inside a single-quoted string in a POSIX-compliant shell.

---

What's different? Lots:

• We're generating the commands we want to run as arrays , and letting Python do the work of converting those arrays to strings where necessary. This avoids shell injection attacks, a very common class of security vulnerability.
• Because we're generating lists ourselves, we can change how we quote each one: We can use f-strings when it's appropriate to do so, raw strings when it's appropriate, etc.
• We aren't passing ~ to the remote server: It's redundant and unnecessary because ~ is the default place for a SSH session to start; and the security precautions we're using (to prevent values from being parsed as code by a shell) prevent it from having any effect (as the replacement of ~ with the active value of HOME is not done by sed itself, but by the shell that invokes it; because we aren't invoking any local shell at all, we also needed to use os.path.expanduser to cause the ~ in ~/.ssh/my-ssh-key to be honored).
• Because we aren't using a raw string, we need to double the backslashes in \\b to ensure that they're treated as literal rather than syntactic by Python.
• Critically, we're never passing data in a context where it could be parsed as code by any shell, either local or remote .