在 python (swig) 中将 mmap 转换为 ctypes void **

Question

I have used swig to generate a wrapper of a library in C.

I need to replicate a code in C in python, but I don't know how to go from mmap to void **

in C:

char *image_buf;
...
axidma_fd = open("/dev/axidma", O_RDWR|O_EXCL);
...
image_buf = mmap(NULL, image_byte_size, PROT_READ|PROT_WRITE,
                      MAP_SHARED, axidma_fd, 0);
...
trans.num_frame_buffers = 1;
trans.frame_buffers = (void **)&image_buf;

in python

axidma_fd = os.open("/dev/axidma", os.O_RDWR|os.O_EXCL)
...
image_buf_1 = mmap.mmap(vdma_fd,188, mmap.MAP_SHARED,mmap.ACCESS_WRITE| mmap.ACCESS_READ);
...
# Convert mmap to c_char 
char_pointer = c_char.from_buffer(image_buf_1) ???? (but type is c_char not pointer)
# Double void
trans.frame_buffers = ??

How do I convert an mmap to a pointer?

How do I get the double pointer?

Would it be more efficient to do it in python directly or by instantiating the libc library? for example like this

libc = ctypes.CDLL("libc.so.6")
image_mmap = ctypes.c_void_p(libc.mmap(0,188,mmap.ACCESS_WRITE| mmap.ACCESS_READ,mmap.MAP_SHARED,vdma_fd)

Thank you so much

EDITED 1 It explains me a little badly

if I do:

image_buf_1 = mmap.mmap(vdma_fd,188, mmap.MAP_SHARED,mmap.ACCESS_WRITE| mmap.ACCESS_READ)
image_map = c_void_p.from_buffer(image_buf_1)

image_map is None

>>> image_map
c_void_p(None)

On the other hand if I do

libc = ctypes.CDLL("libc.so.6")
image_mmap = ctypes.c_void_p(libc.mmap(0,188,mmap.ACCESS_WRITE| mmap.ACCESS_READ,mmap.MAP_SHARED,vdma_fd))

image_mmap is a valid pointer.

>>> image_mmap
c_void_p(18446744073709551615)

But when trying to assign it to

trans.frame_buffers = image_mmap

report:

trans.frame_buffers=image_mmap
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: in method 'axidma_video_transaction_frame_buffers_set', argument 2 of type 'void **'

I need convert mmap to void **

Answer 1

Listing [Python.Docs]: mmap - Memory-mapped file support .

Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) when working with CTypes .

Working with double (or multiple) pointers (that store array like data - like images are) is a bit tricky in CTypes (as seen in [SO]: C++ & Python: Pass and return a 2D double pointer array from python to c++ (@CristiFati's answer) ).
Below is an example that uses arrays.

code00.py :

#!/usr/bin/env python

import ctypes as ct
import mmap
import os
import sys


#CharPtrPtr = ct.POINTER(ct.POINTER(ct.c_char))


def main(*argv):
    h = 3
    w = 4
    fname = "file.txt"
    with open(fname, mode="wb") as f:
        f.write(b"\x00" * h * w)
    fd = os.open(fname, os.O_RDWR | os.O_CREAT)
    buf = mmap.mmap(fd, h * w, mmap.MAP_SHARED, mmap.ACCESS_WRITE | mmap.ACCESS_READ)
    CharArrArr = ct.c_char * w * h
    img = CharArrArr.from_buffer(buf)
    for i in range(h):
        for j in range(w):
            img[i][j] = 0x41 + 2 * i * w + j
    del img
    buf.close()
    os.close(fd)
    with open(fname) as f:
        print("File contents:", f.read())


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output :

 (qaic-env) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q073614606]> python./code00.py Python 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] 064bit on linux File contents: ABCDIJKLQRST Done.

Looking at the edit, it seems that I didn't get it right. Why do you need the void** pointer? What are you doing with it (what is trans )? Please share the code, not just scattered snippets.
If i got things right you need to get mmap 's return value (as void* ), and pass its address (which is a void** ) further. You can do it (using snippets from your code):

image_buf_1 = mmap.mmap(vdma_fd, 188, mmap.MAP_SHARED, mmap.ACCESS_WRITE | mmap.ACCESS_READ)
image_map = ctypes.c_void_p.from_buffer(image_buf_1)
trans.frame_buffers = ctypes.byref(image_mmap)  # or ctypes.pointer(image_mmap)

The other problem ( NULL pointer): You're passing fd 0 (which is StdIn ) to libc.mmap (also check 2 ^nd URL for your hidden bug), while for mmap.mmap you're using the "real" descriptor, So I assume for that descriptor, memory couldn't be properly mapped (you should check mmap results in both cases ( image_buf_1 )).

Answer 2

Thanks in part to this answer: ctypes reference double pointer

try ctypes.byref() . Sample code:

import ctypes
import mmap
import os

FILENAME=#<your filename>
FILELENGTH=#<your file's length>

fd = os.open(FILENAME, os.O_RDWR|os.O_EXCL)
image_buf_1 = mmap.mmap(fd, FILELENGTH, mmap.MAP_SHARED|mmap.ACCESS_WRITE|mmap.ACCESS_READ)

image_map = ctypes.c_void_p.from_buffer(image_buf_1)
print(f"{image_map.value:x}")
print(f"{ctypes.byref(image_map)}")      # this is the pointer you're looking for

os.system(f'cat /proc/$PPID/maps | grep {FILENAME}')    # verify the pointer value

os.close(fd)

Sorry, I don't know exactly what you mean by "the double pointer"; mmap returns a void*, represented here by ctypes.byref(image_map).

As for what's more efficient, you can of course profile, but usually your investment comes from doing the I/O rather than setting up the I/O (I'm assuming that ctypes.DLL is implemented via dlopen() which is pretty lightweight for an already-loaded library). Unless you're opening a very large number of files, I'd guess you want to choose your method based on whether you want to use the data in Python (you may need the Python mmap object) or not.

--note: I briefly tried the libc method you cited, but mmap returned -1, Invalid argument. I didn't look into what I was doing wrong there.