Memory protection in Linux kernel

Question

When the shared library is mapped into the memory, Linux kernel will assign virtual memory areas to this memory region and mark their permissions respectively. But we know that there is no executable bit in the page table entry in x86 arch. If there is a call instruction like "call *edx" in the program to invoke function in the shared library, how can the Linux kernel know whether the target address is executable or not? Will it cause general protection fault if the permission is not compatible in the vma list?

Answer 1

It can't.

On an architecture (/operating mode) where there is no execute page permission distinct from the read permission, the kernel would not be able to discover a supposedly illegal address execution as a result of a fault detected by the MMU.

Answer 2

You're right that in theory the kernel could decide based on the finer-grained permissions on the memory region object, but such a decision making procedure would have to be in the page fault handler, which (I suppose) would make routine memory accesses very expensive.

The kernel instead employs these simplification rules on x86:

• read access right always implies execute access right
• write access right always implies read access right

Source: Understanding Linux Kernel, 1st edition, page 205