How can I ask Mac OS to allocate memory in a specific address range?

Question

I am writing a Mac OS 9 "compatibility layer" for Mac OS X because I was struck with nostalgia recently, and because all current solutions require you to run Classic inside a virtual machine that doesn't support everything it should to run the stuff I want to use.

To achieve that goal, I implemented a PEF executable loader, a PowerPC interpreter that can bridge to native code, and some Mac OS 9 libraries. The whole thing is kind of working: there are some bugs in the interpreter, but that was expected and I'm working on that.

My biggest issue so far is that PowerPC programs are 32-bits, so pointers need to be 32-bits. To satisfy that constraint, currently I'm compiling for i386 only. However, as I'm trying to build around that core, it's becoming more and more constraining (for instance, you can't use ARC with 32-bits Cocoa applications). Not to mention that it's not super-safe to let the PowerPC code access everything the host process can.

I've designed the "platform" with an eventual switch to 64-bits in mind: the interpreter expects a base address and all PowerPC pointers are offset by that base address. (Having a map that translates PowerPC addresses to native addresses is out of question, for performance and design reasons.)

Since a 64-bits address space has enough room to host four billions independent 32-bits address spaces, this sounds like a good way to go for me.

But I still have a problem: I need to be sure that I'll be able to allocate memory inside that address range, because it would be impossible for the PPC interpreter to access anything outside of it.

There are two solutions I am thinking of right now:

• Find out if I can ask Mac OS to allocate something inside a specific address range. I know I can use mmap , but mmap will ask for allocations in page multiples, which seems very wasteful to me since I'd need a full page for each allocation, no matter how small it is (though that could actually be okay considering that Classic-era Macs had very little memory compared to modern computers);
• Use mmap to reserve a full 0x100000000 bytes with PROT_NONE , then mprotect pages on demand to actually allocate memory when it's needed, and put them back to PROT_NONE when they're not useful anymore. It looks good on paper, but it means I have to implement a malloc replacement.

So, what should I do? Is there a built-in mechanism that will let me try to allocate memory, malloc -style, inside a specific address range? Otherwise, is there a good, readable, open-source implementation of malloc I could base myself on?

Answer 1

I'm not aware of a built-in way to do this, but it is doable with a bit of work. One approach would be to create a custom malloc zone, and then use malloc_zone_* versions of the usual malloc functions when allocating any memory that needs to be visible to your PPC code. Your custom zone would need a malloc implementation, but you can pick from any number of open-source ones (eg tcmalloc). It would also need to be wired up to use something like vm_allocate with a hint address, to ensure you get allocations in the exact range you need.

Answer 2

So, I've considered this issue myself for a very similar (unpublished) project, and trying to use this 64-bit approach (or, indeed any sort of big-endian memory space on x86!) rapidly runs into a few nasty issues.

1. While you can point native APIs at emulated structures, you can't force them to make all their own allocations within that arena. You can provide your own implementations of all the most obvious ones, like _NewPointer , but internal allocations by other functions (like, say, _NewWindow with nil wStorage ) are largely out of your control.

2. System structures (like a Rect, for instance) allocated by emulated code will be big-endian, but the OS will want them little-endian. There's no easy way to get around this besides maybe byte-swapping everything on the way in and out of native functions, and that seems hugely error-prone. (Another approach I considered here was to just make the emulated core run as little-endian, but some programs are, unfortunately, sensitive to this change.)

To be clear, I love the hell out of the idea of a Classic-to-OS-X API translator. But I'm pretty sure that it's impractical -- given that Apple ended using a VM for Classic on PPC, I suspect that they determined that a translation approach wasn't going to work.