从 Python ctypes 调用 Go 字符串 function 导致段错误

Question

I have a module called test.go that contains two simple Go functions which accept string types:

package main

import (
  "fmt"
  "C"
)

//export TestConcat
func TestConcat(testArg string, testArg2 string) (string) {
  retval := testArg + testArg2
  return retval
}

//export TestHello
func TestHello(testArg string) {
  fmt.Println("%v\n", testArg)
}


func main(){}

I compile it as a shared library with go build -o test.so -buildmode=c-shared test.go

Then I have a Python module called test.py

import ctypes

from ctypes import cdll


test_strings = [
    "teststring1",
    "teststring2"
]

if __name__ == '__main__':
    lib = cdll.LoadLibrary("./test.so")
    lib.TestConcat.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p]
    lib.TestHello.argtypes = [ctypes.c_wchar_p]
    for test_string in test_strings:
        print(
            lib.TestConcat("hello", test_string)
        )
        lib.TestHello(test_string)

Then I run test.py and get a nasty segfault

runtime: out of memory: cannot allocate 279362762964992-byte block (66781184 in use)
fatal error: out of memory

I've tried wrapping the arguments in ctypes.c_wchar_p to no avail.

What am I doing wrong here? And specifically, how does one interact with Go functions that accept string arguments in Python?

Answer 1

The Go's string type is actually something like

type string {
    ptr *byte
    size int
}

so that is what the Test{Hello|Concat} actually expect—not a pair of pointers but a pair of struct -typed values.
In other words, cgo performs just enough magic to gateway calls from Go to C and back, but it does not perform automatic conversions of values.

You have two options:

• Explicitly work with this from your ctypes bindings, if possible.
  When compiling your package, cgo generates a header file which contains a C definition for the struct representing a Go string; you could use it right away.

• Make the functions exported to C compatible with the C's "type system".
  For this, cgo offers helper functions C.CString and C.GoString .
  Basically, you can define your API like this:

   func TestHello(a, b *C.char) *C.char { testArg1, testArg2:= C.GoString(a), C.GoString(b) return C.CString(testArg + TestArg2) }

  Note few caveats here:

  • Both of these helpers copy the memory of their argument, so the silly example above would work just fine but it would first duplicate the memory blocks pointed to by a and b , then eat up twice as much memory to produce the concatenated string and then copy the memory of the resulting string once again to produce the returned pointer.
    IOW, this approach is fine if you're trying to export to C some big chunk of Go code so that these allocations are dwarfed by whatever that chunk does.
  • Using *C.char is the same as *char in C, so the string is expected to be NUL-terminated; if it's not, use C.GoStringN .
  • Every memory block allocated by C.CString have to be freed by a call to C.free . And here's a twist: C.free is basically a thin shim to call free() from the linked in libc , so if you can guarantee the complete product (the code fully loaded into memory and (inter)linked using the dymanic linker) has only a single copy of libc linked in, you can call free() from the non-Go code on the memory blocks produced by calls to C.Cstring in the Go code.

A few more random pointers:

• I'm not well-versed in Python's ctypes but I'd speculate using ctypes.c_wchar_p is not correct: in C (and C++, FWIW) wchar_t is a type to denote a single fixed-sized "wide character", which is usually a UCS-2/UTF-16 code point, and Go's strings are not composed of these—they may contain arbitrary bytes, and when they are used to contain Unicode text, they are encoded using UTF-8 which is a multi-byte ecoding (a single Unicode code point may be represented by 1 to 4 bytes in the string).
  In either case, wchar_t cannot be used for UTF-8 (and actually many seasoned devs beleive it's an abomination ).
• Please read the docs on cmd/cgo completely before embarking on this project. Really, please do!