How do you convert a char * with 0-value bytes into a python string?

Question

Using the ctypes module I can easily import a POINTER(c_char) or a c_char_p type into python, but neither of these provides a way to end up with a python string that contains zero value bytes.

c_char_p is zero terminated, meaning that a char * array from C is terminated at the first zero value.

POINTER(c_char) is the recommended way of importing binary data that can have 0 values, but there doesn't seem to be a way to directly convert this into a python string.

I can do this:

pixels = clibblah.get_pixels()
a = ""
for i in range(0, clibblah.get_pixel_length()):
    a += pixels[i]

...but this 1) doesn't seem very pythony, and 2) takes forever (converting a 640x480 block of pixel data takes about 2 seconds on my mac).

I've seen a bunch of questions regarding this on stack overflow, but darned if I can see one that isn't either people going "why do you need to do that?" or "c_char_p will do what you want" (it doesn't, as I've described above).

The only credible advice I've seen is to use the c api PyString_FromStringAndSize, as recommended here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/FAQ.html

Can't really see how that helps though, because afaik that's a cython feature, not a python one.

For the interested, the reason I need to do this is I'm working with panda3d and a kinect, and the kinect c api provides an array of unsigned char * values and the panda3d api lovingly provides a setPixels() call that only takes a python string as an argument.

Answer 1

As you said, use a POINTER(c_char) to get a pointer to the array of binary data. To put that together into a string, you can just take a slice of it, since array indexing works as expected with ctypes pointers:

clibblah = ctypes.cdll.LoadLibrary('clibblah.dylib')
get_pixels = clibblah.get_pixels
get_pixels.restype = ctypes.POINTER(ctypes.c_char)

pixels = get_pixels()
num_pixels = clibblah.get_pixel_length()

# Slice the ctypes array into a Python string
a = pixels[:num_pixels]

Answer 2

There are a few different methods. I like ctypes.string_at because it isn't finicky: it works regardless of whether you supply a c_char_p type, or a pointer-to- c_char , or a void pointer type, or even just an int address.

s = b'hello\x00world' # create a string containing null bytes
sz = len(s)
from ctypes import *

p = c_char_p(s) # obtain a pointer of various types
p2 = cast(p,POINTER(c_char))
address = cast(p,c_void_p).value

print p.value # by default it is interpreted as null-terminated

print p2[:sz] # various methods of explicitly specifying the full length
print string_at(p,size=sz)
print (c_char * sz).from_address(address).raw

Answer 3

I don't know what the best answer to the main question is, but here are a few comments about how PyString_FromStringAndSize could be used to accomplish what you want.

PyString_FromStringAndSize is part of the Python C API: http://docs.python.org/c-api/string.html

That means you can use this to

• Write a Python module in C/C++ in which you define a new Python data type your C-derived strings-with-null-characters
• You could define that data type so it provides a Python constructor that accepts arguments that somehow contain a pointer to the C-string in question. If nothing helps, the argument the constructor accepts could be a c_void_p from cytpes.
• The constructor you define (in C/C++) would have to store a pointer to the C-string in a member variable. It might also make a copy and/or increase reference counts etc. Since the constructor is written in C/C++, anything that is possible there is possible in that constructor.

You would have to build this into a.dll/.pyd library and then can import it into any Python code.

Admittedly, a rather complicated procedure. Hopefully somebody else suggests a simpler way, perhaps based on ctypes directly.