Is there a graceful way to pass large byte slice or io.Reader to c through cgo without memory copy?

Question

I have ac program that needs a lot of data to work.

Suppose I have a byte slice of 4GB in go, just loaded from a large file, or internet, or socket.

Then I want to pass it to cgo. If I pass a pointer to the byte buffer, the buffer is still in go, because the buffer can be garbage collected, and thus create segment fault in C side. And io.Reader cannot be simply passed to cgo beacuse it's too general.

I know I can pass the filepath, or write internet request code in C to avoid a cgo bytes pass. But that is not convenient, I still want to keep the advantage of convenience of golang.

So how can we gracefully handle large bytes pass in cgo? If there is no another option, I want to make sure there is no another option.

Answer 1

If I pass a pointer to the byte buffer, the buffer is still in go, because the buffer can be garbage collected, and thus create segment fault in C side.

You can safely pass a Go pointer to C, as long as the C function does not refer to that pointer any more after the call returns. The Go compiler will pin the allocation in place for the duration of the C call.

See https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers for the precise details.