there is any way to open and read a file over a SSH connection?

Question

I have an access to some server where there is a lot of data. I can't copy the whole of data on my computer.

I can't compile on the server the program I want because the server doesn't have all libs I need.

I don't think that the server admin would be very happy to see me coming and asking to him to install some libs just for me...

So, I try to figure if there is a way to open a file like with,

FILE *fopen(const char *filename, const char *mode);

or

void std::ifstream::open(const char* filename,  ios_base::openmode mode = ios_base::in);

but over a SSH connection. Then reading the file like I do for usual program.

both computer and server are running linux

Answer 1

I assume you are working on your Linux laptop and the remote machine is some supercomputer.

First non-technical advice: ask permission first to access the data remotely. In some workplaces you are not allowed to do that, even if it technically possible.

You could sort-of use libssh for that purpose, but you'll need some coding and read its documentation.

You could consider using some FUSE file system (on your laptop), eg some sshfs ; you would then be able to access some supercomputer files as /sshfilesystem/foo.bar ). It is probably the slowest solution, and probably not a very reliable one. I don't really recommend it.

You could ask permission to use NFS mounts.

Maybe you might consider some HTTPS access (if the remote computer has it for your files) using some HTTP/HTTPS client library like libcurl (or the other way round, some HTTP/HTTPS server library like libonion )

And you might (but ask permission first!) use some TLS connection (eg start manually a server like program on the remote supercomputer) perhaps thru OpenSSL or libgnutls

At last, you should consider installing (ie asking politely the installation on the remote supercomputer) or using some database software (eg a PostgreSQL or MariaDB or Redis or MongoDB server) on the remote computer and make your program become a database client application ...

BTW, things might be different if you access a few dozen of terabyte sized files in a random access (each run reading a few kilobytes inside them), or a million files, of which a given run access only a dozen of them with sequential reads, each file of a reasonable size (a few megabytes). In other words, DNA data, video films, HTML documents, source code, ... are all different cases!

Answer 2

Well, the answer to your question is no , as already stated several times (unless you think about implementing ssh yourself which is out of scope of sanity).

But as you also describe your real problem, it's probably just asking the wrong question, so -- looking for alternatives:

Alternative 1

Link the library you want to use statically to your binary. Say you want to link libfoo statically:

• Make sure you have libfoo.a (the object archive of your library) in your library search path. Often, development packages for a library provided by your distribution already contain it, if not, compile the library yourself with options to enable the creation of the static library

• Assuming the GNU toolchain, build your program with the following flags: -Wl,-Bstatic -lfoo -Wl,-Bdynamic (instead of just -lfoo )

Alternative 2

Create your binary as usual (linked against the dynamic library) and put that library ( libfoo.so ) eg in ~/lib on the server. Then run your binary there with LD_LIBRARY_PATH=~/lib ./a.out .

Answer 3

You can copy parts of file to your computer over SSH connection:

• copy part of source file using dd command to temporary file
• copy temporary file to your local box using scp or rsync

You can create a shell script to automate this if you need to do that multiple times.

Answer 4

Instead of fopen on a path, you can use popen on an ssh command. (Don't forget that FILE * streams obtained from popen are closed with pclose and not fclose ).

You can simplify the interface by writing a function which wraps popen . The function accepts just the remote file name, and then generates the ssh command to fetch that file, properly escaping everything, like spaces in the file name, shell meta-characters and whatnot.

FILE *stream = popen("ssh user@host cat /path/to/remote/file", "r");

if (stream != 0) {
   /* ... */
  pclose(stream);
}

popen has some drawbacks because it processes a shell command. Because the argument to ssh is also a shell command that is processed on the remote end, it raises issues of double escaping: passing a command through as a shell command.

To do something more robust, you can create a pipe using pipe , then fork and exec* the ssh process, installing the write end of the pipe as its stdout, and use fdopen to create a FILE * stream on the reading end of the pipe in the parent process. This way, there is accurate control over the arguments which are handed to the process: at least locally, you're not running a shell command.

Answer 5

You can't directly(1) open a file over ssh with fopen() or ifstream::open. But you can leverage the existing ssh binary. Simply have your program read from stdin, and pipe the file to it via ssh:

ssh that_server cat /path/to/largefile | ./yourprogram

(1) Well, if you mount the remote system using sshfs you can access the files over ssh as if they were local.