How to transfer data into remote file over SFTP without storing data in a local file in Linux Bash?

Question

I need to be able to transfer data from memory into a remote file through SFTP. I originally had this working through SSH, and while working, discovered that I don't have SSH access to the remote location, only SFTP access.

Here is an example of the original SSH code I had:

echo "secret_data" | ssh root@localhost cat > secret_file;

This is exactly what I need, but in some kind of form of:

sftp root@localhost put $secret_data secret_file;

In principal, I need the data to never be stored on a file on the local machine, and dealt with entirely from memory.

Any replies appreciated. Thanks.

Answer 1

The multi-protocol client lftp explicitly supports reading content from a non-seekable file descriptor:

#!/bin/bash
#      ^^^^ some features used here are not present in /bin/sh

lftp \
  -u remote_username                 \
  -e 'put -o /tmp/secret /dev/stdin' \
  sftp://remote_host                 \
  < <(printf '%s' "$secret_data")

Note the use of <() as opposed to <<< (the latter can, in some situations, be implemented via writing a temporary file; the former will be a /dev/fd -style redirection on modern Linux, or may be implemented with a FIFO on some other platforms).

Answer 2

Create tmpfs partition in /etc/fstab (not stored on drive, only in memory), store the file there and then use your described method if you really want to avoid storing the file into your hard drive.

I have set up this way /tmp and /var/log to avoid writing all over the SSD drive:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
none    /tmp/       tmpfs   size=15%    0   0

Answer 3

If it's permissible to use your own, hacked copy of sftp, you can use

echo "my secret data 2" | (exec 6<&0 ; ( echo put  /proc/self/fd/6 /tmp/secret | sftp user@remote_host))

The exec redirects the stdin, which transports your secret, to another file descriptor, in this case, 6. The second echo issues the sftp command to execute. It uses the /proc/ magic file system, the /proc/self redirection to /proc/<pid> of the process that opens it, and the proc/<pid>/fd/6 name of the file descriptor 6, and copies the data that it reads from there to a file on your remote host.

It would be much easier if you could use a hacked version of scp, this would read

echo "my secret data 2" | scp /proc/self/fd/0 user@remote_host:/tmp/secret

Now for the hack: sftp and scp make sure that the local file is a regular file, but the /proc/self/fd/... file descriptors are pipes. You need to disable the checks in the source code.

For sftp, you would modify file sftp-client.c: Find all occurences of S_ISREG(...) and replace them with 1

This is quick and dirty and might leave you open to security vulnerabilities if you do not check openssh security messages regularly and recompile if necessary. A better way would be to use a scripting language with a well maintained sftp library and use that.