Mac OS X: recvmsg returns EMSGSIZE when sending fd's via Unix domain datagram socket

Question

I have a piece of code that uses Unix domain sockets and sendmsg/recvmsg to send fd's between two processes. This code needs to run on both Linux and Mac (it is complied separately for both platforms). I'm using SOCK_DGRAM (datagram) sockets.

I send one fd at a time in my code. On Mac, after sending a couple of fd's succesfully this way, recvmsg() fails with an EMSGSIZE. According to the manpage for recvmsg, this can only happen if msg->msg_iovlen <=0 or >= a constant which is 2048 on Mac. In my code, I've pegged msg_iovlen to 1 always, I verified this on the sender and receiver, and also from reading the message header right after recvmsg() faults. This same code works fine on Linux.

Another possibility, from looking at the XNU kernel source, is that the receiver could have run out of fd's, but I've only sent 4 or 5 fd's before the error happens so there should be plenty of fd's left.

If I don't send fd's and only send data, this error does not occur.

Here's what the code that's packing the control message looks like:

// *obj is the fd, objSize is sizeof(*obj)
// cmsg was allocated earlier as a 512 byte buffer

cmsgLength = CMSG_LEN(objSize);

cmsgSpace = CMSG_SPACE(objSize);

cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_RIGHTS;
cmsg->cmsg_len = cmsgLength;

memcpy(CMSG_DATA(cmsg), obj, objSize);

msg->msg_control = cmsg;
msg->msg_controllen = cmsgSpace;

And here's the receiver:

msg = (struct msghdr *)pipe->msg;
iov = msg->msg_iov;
iov->iov_base = buf;
iov->iov_len = size;

// msg->msg_control was set earlier
msg->msg_controllen = 512;

return recvmsg(sockFd, msg, 0);

Any clues? Thanks in advance

Answer 1

Are you actually using the cmsg stuff that you are receiving? I notice that you set msg_controllen to 512. What have you set msg_flags to?

Would you be able to try the same thing out with the following one addition.

msg = (struct msghdr *)pipe->msg;

memset (msg, 0, sizeof(msghdr)); /* added this */

iov = msg->msg_iov;
iov->iov_base = buf;
iov->iov_len = size;

// msg->msg_control was set earlier
msg->msg_controllen = 512;

return recvmsg(sockFd, msg, 0);