OSX Shared memory access not permitted from GUI Apps

Question

I have a dylib on OSX I'm injecting through DYLD_INSERT_LIBRARIES and it calls

shm_open("name", O_RDWR);

The shared memory was set up in another process using

shm_open("name", O_RDWR|O_CREAT, 0777);

Communication from injected dyld to external app is successful when run in /bin/bash and other command line programs. Testing in GUI apps such as Calculator or Preview yield an error when shm_open is called from the injected dylib. The error is "Operation not permitted".

Does anyone know the reason behind this?

Answer 1

Both of the applications that you've mentioned — Calculator and Preview — are sandboxed. One of the things this prevents them from doing is accessing shared memory (outside of certain specific circumstances which do not apply to these applications). See Apple's documentation " App Sandbox Design Guide: App Sandbox in Depth " for details.

If you need to communicate with arbitrary sandboxed processes, you will need to use an alternate method to do so (possibly XPC, although I'm not very familiar with that). If you just want to test dylib injection in GUI processes, try a non-sandboxed application like Stickies (as of Mac OS 10.9).