Can I adapt a function that writes to disk to write to memory

Question

I have third-party library with a function that does some computation on the specified data, and writes the results to a file specified by file name:

int manipulateAndWrite(const char *filename,
                       const FOO_DATA *data);

I cannot change this function, or reimplement the computation in my own function, because I do not have the source.

To get the results, I currently need to read them from the file. I would prefer to avoid the write to and read from the file, and obtain the results into a memory buffer instead.

Can I pass a filepath that indicates writing to memory instead of a filesystem?

Answer 1

Yes, you have several options, although only the first suggestion below is supported by POSIX. The rest of them are OS-specific, and may not be portable across all POSIX systems, although I do believe they work on all POSIXy systems.

• You can use a named pipe (FIFO), and have a helper thread read from it concurrently to the writer function.

  Because there is no file per se, the overhead is just the syscalls (write and read); basically just the overhead of interprocess communication, nothing to worry about. To conserve resources, do create the helper thread with a small stack (using pthread_attr_ etc.), as the default stack size tends to be huge (on the order of several megabytes; 2*PTHREAD_STACK_SIZE should be plenty for helper threads.)

  You should ensure the named pipe is in a safe directory, accessible only to the user running the process, for example.
  

• In many POSIXy systems, you can create a pipe or a socket pair, and access it via /dev/fd/N , where N is the descriptor number in decimal. (In Linux, /proc/self/fd/N also works.) This is not mandated by POSIX, so may not be available on all systems, but most do support it.

  This way, there is no actual file per se, and the function writes to the pipe or socket. If the data written by the function is at most PIPE_BUF bytes, you can simply read the data from the pipe afterwards; otherwise, you do need to create a helper thread to read from the pipe or socket concurrently to the function, or the write will block.

  In this case, too, the overhead is minimal.
  

• On ELF-based POSIXy systems (basically all), you can interpose the open() , write() , and close() syscalls or C library functions.

  (In Linux, there are two basic approaches, one using the linker --wrap , and one using dlsym() . Both work fine for this particular case. This ability to interpose functions is based on how ELF binaries are linked at run time, and is not directly related to POSIX.)

  You first set up the interposing functions, so that open() detects if the filename matches your special "in-memory" file, and returns a dedicated descriptor number for it. (You may also need to interpose other functions, like ftruncate() or lseek() , depending on what the function actually does; in Linux, you can run a binary under ptrace to examine what syscalls it actually uses.)

  When write() is called with the dedicated descriptor number, you simply memcpy() it to a memory buffer. You'll need to use global variables to describe the allocated size, size used, and the pointer to the memory buffer, and probably be prepared to resize/grow the buffer if necessary.

  When close() is called with the dedicated descriptor number, you know the memory buffer is complete, and the contents ready for processing.
  

• You can use a temporary file on a RAM filesystem. While the data is technically written to a file and read back from it, the operations involve RAM only.

  You should arrange for a default path to one to be set at compile time, and for individual users to be able to override that for their personal needs, for example via an environment variable ( YOURAPP_TMPDIR ?).

  There is no need for the application to try and look for a RAM-based filesystem: choices like this are, and should be, up to the user. The application should not even care what kind of filesystem the file is on, and should just use the specified directory.

Answer 2

You could not use that library function. Take a look at this on how to write to in-memory files: Is it possible to create a C FILE object to read/write in memory