How does making the virtual address contiguous in physical address zone improve performance?

Question

Recently I am reading code about hugepage in dpdk(dpdk.org). I see the code makes the virtual address contiguous in physical address zone on purpose. Specifically, it first checks if there exists physically contiguous zone in hugepages and map the physically contiguous zone into contiguous virtual address. How does this improve the performance?

The source code says:

To reserve a big contiguous amount of memory, we use the hugepage feature of linux. For that, we need to have hugetlbfs mounted. This code will create many files in this directory (one per page) and map them in virtual memory. For each page, we will retrieve its physical address and remap it in order to have a virtual contiguous zone as well as a physical contiguous zone.

Why is this remapping necessary?

Answer 1

map the physically contiguous zone into contiguous virtual address. How does this improve the performance?

DPDK needs both physical and virtual addresses. The virtual address is used normally, to load/store some data. The physical address is necessary for the userspace drivers to transfer data to/from devices.

For example, we allocate a few mbuf s with virtual addresses 0x41000 , 0x42000 and 0x43000 . Then we fill them with some data and pass those virtual addresses to the PMD to transfer.

The driver has to convert those virtual addresses to physical. If physical pages are mapped to virtual address space noncontiguous, to convert virtual to physical addresses we need to search through all the mappings. For example, virtual address 0x41000 might correspond to physical 0x81000 , 0x42000 corresponds to 0x16000 , and 0x43000 — to 0x64000 .

The best case of such a search is one memory read, the worst case — a few memory reads for each buffer .

But if we are sure that both virtual and physical addresses of a memory zone are contiguous, we simply add an offset to the virtual address to get the physical and vice versa. For example, virtual 0x41000 corresponds to 0x81000 , virtual 0x42000 to physical 0x82000 , and 0x43000 — 0x83000 .

The offset we know from the mapping. The worst case of such a translation is one memory read per all the buffers in a burst , which is a huge improvement for the translation.

Why is this remapping necessary?

To map a huge page to a virtual address space an mmap system call is used. The API of the call allows to specify the fixed virtual address for the huge page to be mapped. This allows to map huge pages one after another creating a contiguous virtual memory zone . For example, we can mmap a huge page at the virtual address 0x200000 , the next one at the virtual address 0x400000 and so on.

Unfortunately, we don't know physical addresses of the huge pages until they are mapped. So at the virtual address 0x200000 we might map the physical address 0x800000 , and at the virtual address 0x400000 — the physical 0x600000 .

But once we mapped those huge pages for the first time, we know both physical and virtual addresses. So all we need to do is to remap them in the correct order: at virtual address 0x1200000 we map physical 0x600000 , and at 0x1400000 — physical 0x800000 .

Now we have a virtually and physically contiguous memory zone starting at the virtual address 0x1200000 and physical address 0x600000 . So to convert virtual to physical addresses in this memory zone we just need to subtract the offset 0x600000 from the virtual address as described previously.

Hope this clarifies a bit the idea of contiguous memory zones and remapping.