生成器理解和列表理解的迭代方式不同

Question

I wrote a function that passes numpy array's into C code using CFFI. It utilizes the buffer protocol and memoryview to pass the data efficiently without copying it. However, this means that you need to pass C-contiguous arrays and ensure that you using the right types. Numpy provides a function numpy.ascontiguous, which does this. So I iterate over the arguments, and apply this function. The implementation below works, and may be of general interest. However, it is slow given the number of times it is called. (Any general comments on how to speed it up would be helpful.)

However, the actual question is when you replace the first list comprehension with a generator comprehension, or if you refactor the code so that np.ascontigous is called in the second one, the pointers passed into the C code no longer point to the start of the numpy array. I think that it is not getting called. I'm iterating over the comprehension and only using the return values, why would using a list comprehension or generator comprehension change anything?

def cffi_wrap(cffi_func, ndarray_params, pod_params, return_shapes=None):
    """
    Wraps a cffi function to allow it to be called on numpy arrays.

    It uss the numpy buffer protocol and and the cffi buffer protocol to pass the 
    numpy array into the c function without copying any of the parameters. 
    You will need to pass dimensions into the C function, which you can do using 
    the pod_params.

    Parameters
    ----------
    cffi_func : c function
        This is a c function declared using cffi. It must take double pointers and 
        plain old data types. The arguments must be in the form of numpy arrays, 
        plain old data types, and then the returned numpy arrays.
    ndarray_params : iterable of ndarrays
         The numpy arrays to pass into the function.
    pod_params : tuple of plain old data
        This plain old data objects to pass in.  This may include for example 
        dimensions.
    return_shapes : iterable of tuples of positive ints
          The shapes of the returned objects.

    Returns
    -------
    return_vals : ndarrays of doubles.
        The objects to be calculated by the cffi_func.

    """

    arr_param_buffers = [np.ascontiguousarray(param, np.float64) 
         if np.issubdtype(param.dtype, np.float)
         else np.ascontiguousarray(param, np.intc) for param in ndarray_params]
    arr_param_ptrs = [ffi.cast("double *", ffi.from_buffer(memoryview(param))) 
        if np.issubdtype(param.dtype, np.float)
        else ffi.cast("int *", ffi.from_buffer(memoryview(param))) 
        for param in arr_param_buffers]

    if return_shapes is not None:

        return_vals_ptrs = tuple(ffi.new("double[" + str(np.prod(shape)) + "]") 
            for shape in return_shapes)
        returned_val = cffi_func(*arr_param_ptrs, *pod_params, *return_vals_ptrs)
        return_vals = tuple(np.frombuffer(ffi.buffer(
              return_val))[:np.prod(shape)].reshape(shape)
              for shape, return_val in zip(return_shapes, return_vals_ptrs))
    else:
        returned_val = cffi_func(*arr_param_ptrs, *pod_params)
        return_vals = None

    if returned_val is not None and return_vals is not None:
        return_vals = return_vals + (returned_val,)
    elif return_vals is None:
       return_vals = (returned_val,)

    if len(return_vals) == 1:
        return return_vals[0]
    else:
       return return_vals

Answer 1

I'm just guessing, but the error could come from keepalives: with arr_param_buffers a list comprehension, as in your posted code, then as long as this local variable exists (ie for the whole duration of cffi_wrap()), all the created numpy arrays are alive. This allows you to do ffi.from_buffer(memoryview(...)) on the next line and be sure that they are all pointers to valid data.

If you replace arr_param_buffers with a generator expression, it will generate the new numpy arrays one by one, call ffi.from_buffer(memoryview(param)) on them, and then throw them away. The ffi.from_buffer(x) returns an object that should keep x alive, but maybe x == memoryview(nd) does not itself keep alive the numpy array nd , for all I know.