Shared memory across processes on Linux/x86_64

Question

I have a few questions on using shared memory with processes. I looked at several previous posts and couldn't glean the answers precisely enough. Thanks in advance for your help.

1. I'm using shm_open + mmap like below. This code works as intended with parent and child alternating to increment g_shared->count (the synchronization is not portable; it works only for certain memory models, but good enough for my case for now). However, when I change MAP_SHARED to MAP_ANONYMOUS | MAP_SHARED, the memory isn't shared and the program hangs since the 'flag' doesn't get flipped. Removing the flag confirms what's happening with each process counting from 0 to 10 (implying that each has its own copy of the structure and hence the 'count' field). Is this the expected behavior? I don't want the memory to be backed by a file; I really want to emulate what might happen if these were threads instead of processes (they need to be processes for other reasons).

2. Do I really need shm_open? Since the processes belong to the same hierarchy, can I just use mmap alone instead? I understand this would be fairly straightforward if there wasn't an 'exec,' but how do I get it to work when there is an 'exec' following the 'fork?'

3. I'm using kernel version 3.2.0-23 on x86_64 (Intel i7-2600). For this implementation, does mmap give the same behavior (correctness as well as performance) as shared memory with pthreads sharing the same global object? For example, does the MMU map the segment with 'cacheable' MTRR/TLB attributes?

4. Is the cleanup_shared() code correct? Is it leaking any memory? How could I check? For example, is there an equivalent of System V's 'ipcs?'

thanks, /Doobs

shmem.h:

#ifndef __SHMEM_H__
#define __SHMEM_H__

//includes

#define LEN 1000
#define ITERS 10

#define SHM_FNAME "/myshm"

typedef struct shmem_obj {
    int count;
    char buff[LEN];
    volatile int flag;
} shmem_t;

extern shmem_t* g_shared;
extern char proc_name[100];
extern int fd;

void cleanup_shared() {
    munmap(g_shared, sizeof(shmem_t));
    close(fd);
    shm_unlink(SHM_FNAME);
}

static inline 
void init_shared() {
    int oflag;

    if (!strcmp(proc_name, "parent")) {
        oflag = O_CREAT | O_RDWR;
    } else {
        oflag = O_RDWR;
    }

    fd = shm_open(SHM_FNAME, oflag, (S_IREAD | S_IWRITE));
    if (fd == -1) {
        perror("shm_open");
        exit(EXIT_FAILURE);
    }

    if (ftruncate(fd, sizeof(shmem_t)) == -1) {
        perror("ftruncate");
        shm_unlink(SHM_FNAME);
        exit(EXIT_FAILURE);
    }

    g_shared = mmap(NULL, sizeof(shmem_t), 
                    (PROT_WRITE | PROT_READ), 
                    MAP_SHARED, fd, 0);
    if (g_shared == MAP_FAILED) {
        perror("mmap");
        cleanup_shared();
        exit(EXIT_FAILURE);
    }
}

static inline 
void proc_write(const char* s) {
    fprintf(stderr, "[%s] %s\n", proc_name, s);
}

#endif // __SHMEM_H__

shmem1.c (parent process):

#include "shmem.h"

int fd;
shmem_t* g_shared;
char proc_name[100];

void work() {
    int i;
    for (i = 0; i &lt ITERS; ++i) {
        while (g_shared->flag);
        ++g_shared->count;
        sprintf(g_shared->buff, "%s: %d", proc_name, g_shared->count);
        proc_write(g_shared->buff);
        g_shared->flag = !g_shared->flag;
    }
}

int main(int argc, char* argv[], char* envp[]) {
    int status, child;
    strcpy(proc_name, "parent");
    init_shared(argv);
    fprintf(stderr, "Map address is: %p\n", g_shared);

    if (child = fork()) { 
        work();
        waitpid(child, &status, 0);
        cleanup_shared();
        fprintf(stderr, "Parent finished!\n");
    } else { /* child executes shmem2 */
        execvpe("./shmem2", argv + 2, envp);
    } 
}

shmem2.c (child process):

#include "shmem.h"

int fd;
shmem_t* g_shared;
char proc_name[100];

void work() {
    int i;
    for (i = 0; i &lt ITERS; ++i) {
        while (!g_shared->flag);
        ++g_shared->count;
        sprintf(g_shared->buff, "%s: %d", proc_name, g_shared->count);
        proc_write(g_shared->buff);
        g_shared->flag = !g_shared->flag;
    }
}

int main(int argc, char* argv[], char* envp[]) {
    int status;
    strcpy(proc_name, "child");
    init_shared(argv);
    fprintf(stderr, "Map address is: %p\n", g_shared);
    work();
    cleanup_shared();
    return 0;
}

Answer 1

1. Passing MAP_ANONYMOUS causes the kernel to ignore your file descriptor argument and give you a private mapping instead. That's not what you want.

2. Yes, you can create an anonymous shared mapping in a parent process, fork, and have the child process inherit the mapping, sharing the memory with the parent and any other children. That obvoiusly doesn't survive an exec() though.

3. I don't understand this question; pthreads doesn't allocate memory. The cacheability will depend on the file descriptor you mapped. If it's a disk file or anonymous mapping, then it's cacheable memory. If it's a video framebuffer device, it's probably not.

4. That's the right way to call munmap(), but I didn't verify the logic beyond that. All processes need to unmap, only one should call unlink.

Answer 2

2b) as a middle-ground of a sort, it is possible to call:

int const shm_fd = shm_open(fn,...);
shm_unlink(fn);

in a parent process, and then pass fd to a child process created by fork()/execve() via argp or envp. since open file descriptors of this type will survive the fork()/execve(), you can mmap the fd in both the parent process and any dervied processes. here's a more complete code example copied and simplified/sanitized from code i ran successfully under Ubuntu 12.04 / linux kernel 3.13 / glibc 2.15:

int create_shm_fd( void ) {
    int oflags = O_RDWR | O_CREAT | O_TRUNC;
    string const fn = "/some_shm_fn_maybe_with_pid";
    int fd;
    neg_one_fail( fd = shm_open( fn.c_str(), oflags, S_IRUSR | S_IWUSR ), "shm_open" );
    if( fd == -1 ) { rt_err( strprintf( "shm_open() failed with errno=%s", str(errno).c_str() ) ); }
    // for now, we'll just pass the open fd to our child process, so
    // we don't need the file/name/link anymore, and by unlinking it
    // here we can try to minimize the chance / amount of OS-level shm
    // leakage.
    neg_one_fail( shm_unlink( fn.c_str() ), "shm_unlink" );
    // by default, the fd returned from shm_open() has FD_CLOEXEC
    // set. it seems okay to remove it so that it will stay open
    // across execve.
    int fd_flags = 0;
    neg_one_fail( fd_flags = fcntl( fd, F_GETFD ), "fcntl" );
    fd_flags &= ~FD_CLOEXEC;
    neg_one_fail( fcntl( fd, F_SETFD, fd_flags ), "fcntl" );
    // resize the shm segment for later mapping via mmap()
    neg_one_fail( ftruncate( fd, 1024*1024*4 ), "ftruncate" );
    return fd;
  }

it's not 100% clear to me if it's okay spec-wise to remove the FD_CLOEXEC and/or assume that after doing so the fd really will survive the exec. the man page for exec is unclear; it says: "POSIX shared memory regions are unmapped", but to me that's redundant with the general comments earlier that mapping are not preserved, and doesn't say that shm_open()'d fd will be closed. any of course there's the fact that, as i mentioned, the code does seem to work in at least one case.

the reason i might use this approach is that it would seem to reduce the chance of leaking the shared memory segment / filename, and it makes it clear that i don't need persistence of the memory segment.