shm_open：Mac和Linux之间的差异

Question

I have a queue in shared memory. It does work on Linux (kernel 4.3.4), but not on Mac OS X. Are there any differences between how Mac OS X handles shared memory and how linux does, which may explain this?

I get the shared memory via:

int sh_fd = shm_open(shmName, O_RDWR | O_CREAT, 
        S_IROTH | S_IWOTH // others hav read/write permission
        | S_IRUSR | S_IWUSR // I have read/write permission
        );

// bring the shared memory to the desired size
ftruncate(sh_fd, getpagesize());

The queue is very simple as well. Here is the basic struct:

typedef struct {
// this is to check whether the queue is initialized.
// on linux, this will be 0 initially
bool isInitialized;
// mutex to protect concurrent access
pthread_mutex_t access;
// condition for the reader, readers should wait here
pthread_cond_t reader;
// condition for the writer, writers should wait here
pthread_cond_t writer;
// whether the queue can still be used.
bool isOpen;
// maximum capacity of the queue.
int32_t capacity;
// current position of the reader and number of items.
int32_t readPos, items;

// entries in the queue. The array actually is longer, which means it uses the space behind the struct.
entry entries[1];
} shared_queue;

Basically everyone who wants access acquires the mutex, readPos indicates where the next value should be read (incrementing readPos afterwards), (readPos+items) % capacity is where new items go. The only somewhat fancy trick is the isInitialized byte. ftruncate fills the shared memory with zeros if it had length 0 before, so I rely on isInitiualized to be zero on a fresh shared memory page and write a 1 there as soon as I initialize the struct.

As I said, it works on Linux, so I don't think it is a simple implementation bug. Is there any subtle difference between shm_open on Mac vs. Linux which I may not be aware of? The bug I see looks like the reader tries to read from an empty queue, so, maybe the pthread mutex/condition does not work on shared memory in a Mac?

Answer 1

The problem is that PTHREAD_PROCESS_SHARED is not supported on mac.

http://alesteska.blogspot.de/2012/08/pthreadprocessshared-not-supported-on.html

Answer 2

You must set PTHREAD_PROCESS_SHARED on both the mutex and condition variables.

So for a mutex:

pthread_mutexattr_t mutex_attr;
pthread_mutex_t     the_mutex;

pthread_mutexattr_init(&mutex_attr);
pthread_mutexattr_setpshared(&mutex_attr, PTHREAD_PROCESS_SHARED);
pthread_mutexattr(&the_mutex, &mutex_attr);

Basically the same steps for the condition variables, but replace mutexattr with condattr .

If the the pthread_*attr_setpshared functions don't exist or return an error, then it may not be supported on your platform.

To be on the safe side, you might want to set PTHREAD_MUTEX_ROBUST if supported. This will prevent deadlock over the mutex (though not guarantee queue consistency) if a process exits while holding the lock.

EDIT : As an added caution, having a boolean "is initialized" flag is an insufficient plan on its own. You need more than that to really guarantee only one process can initialize the structure. At the very least you need to do:

// O_EXCL means this fails if not the first one here
fd = shm_open(name, otherFlags | O_CREAT | O_EXCL );  
if( fd != -1 )
{
   // initialize here

   // Notify everybody the mutex has been initialized.
}
else
{
    fd = shm_open(name, otherFlags ); // NO O_CREAT

    // magically somehow wait until queue is initialized.
}

Are you sure really need to roll your own queue? Will POSIX message queues (see mq_open man page) do the job? If not, what about one of many messaging middleware solutions out there?

Update 2016-Feb-10: Possible mkfifo based solution

One alternative to implementing your own queue in shared memory is to use an OS provided named FIFO using mkfifo . A key difference between a FIFO and a named pipe is that you are allowed to have multiple simultaneous readers and writers.

A "catch" to this, is that the reader sees end-of-file when the last writer exits, so if you want readers to go indefinitely, you may need to open a dummy write handle.

FIFOs are super easy to use on the command line, like so:

reader.sh

mkfifo my_queue
cat my_queue

write.sh

echo "hello world" > my_queue

Or slightly more effort in C:

reader.c

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>

int main(int argc, char**argv)
{
  FILE * fifo;
  FILE * wfifo;
  int res;
  char buf[1024];
  char * linePtr;

  /* Try to create the queue.  This may belong on reader or writer side
   * depending on your setup. */
  if( 0 !=  mkfifo("work_queue", S_IRUSR | S_IWUSR ) )
  {
    if( errno != EEXIST )
    {
      perror("mkfifo:");
      return -1;
    }
  }

  /* Get a read handle to the queue */
  fifo = fopen("work_queue", "r"); 

  /* Get a write handle to the queue */
  wfifo = fopen("work_queue", "w"); 
  if( !fifo )
  {
    perror("fopen: " );
    return -1;
  }

  while(1)
  {
    /* pull a single message from the queue at a time */
    linePtr = fgets(buf, sizeof(buf), fifo);
    if( linePtr )
    {
      fprintf(stdout, "new command=%s\n", linePtr);
    }
    else
    {
      break;
    }
  } 

  return 0;
}

writer.c

#include <stdio.h>
#include <unistd.h>
int main(int argc, char**argv)
{
  FILE * pipe = fopen("work_queue", "w");
  unsigned int job = 0;
  int my_pid = getpid(); 
  while(1)
  {
    /* Write one 'entry' to the queue */
    fprintf(pipe, "job %u from %d\n", ++job, my_pid);
  }
}