Getting data from ctypes array into numpy

Question

I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and spec

Answer 1

np.ctypeslib.as_array is all you need here.

From an array:

 c_arr = (c_float * 8)()
 np.ctypeslib.as_array(c_arr)

From a pointer

 c_arr = (c_float * 8)()
 ptr = ctypes.pointer(c_arr[0])
 np.ctypeslib.as_array(ptr, shape=(8,))

Answer 2

Creating NumPy arrays from a ctypes pointer object is a problematic operation. It is unclear who actually owns the memory the pointer is pointing to. When will it be freed again? How long is it valid? Whenever possible I would try to avoid this kind of construct. It is so much easier and safer to create arrays in the Python code and pass them to the C function than to use memory allocated by a Python-unaware C function. By doing the latter, you negate to some extent the advantages of having a high-level language taking care of the memory management.

If you are really sure that someone takes care of the memory, you can create an object exposing the Python "buffer protocol" and then create a NumPy array using this buffer object. You gave one way of creating the buffer object in your post, via the undocumented int_asbuffer() function:

buffer = numpy.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), 8*array_length)

(Note that I substituted 8 for np.dtype(float).itemsize. It's always 8, on any platform.) A different way to create the buffer object would be to call the PyBuffer_FromMemory() function from the Python C API via ctypes:

buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory
buffer_from_memory.restype = ctypes.py_object
buffer = buffer_from_memory(y, 8*array_length)

For both these ways, you can create a NumPy array from buffer by

a = numpy.frombuffer(buffer, float)

(I actually do not understand why you use .astype() instead of a second parameter to frombuffer; furthermore, I wonder why you use np.int, while you said earlier that the array contains doubles.)

I'm afraid it won't get much easier than this, but it isn't that bad, don't you think? You could bury all the ugly details in a wrapper function and don't worry about it any more.

Answer 3

Neither of these worked for me in Python 3. As a general solution for converting a ctypes pointer into a numpy ndarray in python 2 and 3 I found this worked (via getting a read-only buffer):

def make_nd_array(c_pointer, shape, dtype=np.float64, order='C', own_data=True):
    arr_size = np.prod(shape[:]) * np.dtype(dtype).itemsize 
    if sys.version_info.major >= 3:
        buf_from_mem = ctypes.pythonapi.PyMemoryView_FromMemory
        buf_from_mem.restype = ctypes.py_object
        buf_from_mem.argtypes = (ctypes.c_void_p, ctypes.c_int, ctypes.c_int)
        buffer = buf_from_mem(c_pointer, arr_size, 0x100)
    else:
        buf_from_mem = ctypes.pythonapi.PyBuffer_FromMemory
        buf_from_mem.restype = ctypes.py_object
        buffer = buf_from_mem(c_pointer, arr_size)
    arr = np.ndarray(tuple(shape[:]), dtype, buffer, order=order)
    if own_data and not arr.flags.owndata:
        return arr.copy()
    else:
        return arr

Answer 4

If you are ok with creating arrays in python, the following example with 2d array works in python3:

import numpy as np
import ctypes

OutType = (ctypes.c_float * 4) * 6
out = OutType()
YourCfunction = ctypes.CDLL('./yourlib.so').voidreturningfunctionwithweirdname
YourCfunction.argtypes = [ctypes.POINTER(ctypes.c_float)]*3, ctypes.POINTER(ctypes.c_float)]*5, OutType]
YourCfunction(input1, input2, out)
out = np.array(out) # convert it to numpy

print(out)

numpy and ctypes versions are 1.11.1 and 1.1.0

Answer 5

Using np.ndarrays as ctypes arguments

The preferable approach is using ndpointer, as mentioned in the numpy-docs.

This approach is more flexible than using, for example, POINTER(c_double), since several restrictions can be specified, which are verified upon calling the ctypes function. These include data type, number of dimensions, shape and flags. If a given array does not satisfy the specified restrictions, a TypeError is raised.

Minimal, Reproducible Example

Calling memcpy from python. Eventually the filename of the standard C-library libc.so.6 needs to be adjusted.

import ctypes
import numpy as np

n_bytes_f64 = 8
nrows = 2
ncols = 5

clib = ctypes.cdll.LoadLibrary("libc.so.6")

clib.memcpy.argtypes = [
    np.ctypeslib.ndpointer(dtype=np.float64, ndim=2, flags='C_CONTIGUOUS'),
    np.ctypeslib.ndpointer(dtype=np.float64, ndim=1, flags='C_CONTIGUOUS'),
    ctypes.c_size_t]
clib.memcpy.restype = ctypes.c_void_p

arr_from = np.arange(nrows * ncols).astype(np.float64)
arr_to = np.empty(shape=(nrows, ncols), dtype=np.float64)

print('arr_from:', arr_from)
print('arr_to:', arr_to)

print('\ncalling clib.memcpy ...\n')
clib.memcpy(arr_to, arr_from, nrows * ncols * n_bytes_f64)

print('arr_from:', arr_from)
print('arr_to:', arr_to)

Output

arr_from: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
arr_to: [[0.0e+000 4.9e-324 9.9e-324 1.5e-323 2.0e-323]
 [2.5e-323 3.0e-323 3.5e-323 4.0e-323 4.4e-323]]

calling clib.memcpy ...

arr_from: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
arr_to: [[0. 1. 2. 3. 4.]
 [5. 6. 7. 8. 9.]]

If you modify the ndim=1/2 arguments of ndpointer to be inconsistent with the dimensions of arr_from/arr_to, the code fails with an ArgumentError.

As the title of this question is quite general, ...

Constructing a np.ndarray from a ctypes.c_void_p result

Minimal, Reproducible Example

In the following example, some memory is allocated by malloc and filled with 0s by memset. Then a numpy array is constructed, to access this memory. Of course the occur some ownership issues, as python will not free memory, which was allocated in c. To avoid memory leaks, one has to free the allocated memory again by ctypes. The copy method can be used for the np.ndarray to acquire ownership.

import ctypes
import numpy as np

n_bytes_int = 4
size = 7

clib = ctypes.cdll.LoadLibrary("libc.so.6")

clib.malloc.argtypes = [ctypes.c_size_t]
clib.malloc.restype = ctypes.c_void_p

clib.memset.argtypes = [
    ctypes.c_void_p,
    ctypes.c_int,
    ctypes.c_size_t]
clib.memset.restype = np.ctypeslib.ndpointer(
    dtype=np.int32, ndim=1, flags='C_CONTIGUOUS')

clib.free.argtypes = [ctypes.c_void_p]
clib.free.restype = ctypes.c_void_p


pntr = clib.malloc(size * n_bytes_int)
ndpntr = clib.memset(pntr, 0, size * n_bytes_int)
print(type(ndpntr))
ctypes_pntr = ctypes.cast(ndpntr, ctypes.POINTER(ctypes.c_int))
print(type(ctypes_pntr))
print()
arr_noowner = np.ctypeslib.as_array(ctypes_pntr, shape=(size,))
arr_owner = np.ctypeslib.as_array(ctypes_pntr, shape=(size,)).copy()
# arr_owner = arr_noowner.copy()


print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))

print('\nfree allocated memory again ...\n')
_ = clib.free(pntr)

print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))

print('\njust for fun: free some python-memory ...\n')
_ = clib.free(arr_owner.ctypes.data_as(ctypes.c_void_p))

print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))

Output

<class 'numpy.ctypeslib.ndpointer_<i4_1d_C_CONTIGUOUS'>
<class '__main__.LP_c_int'>

arr_noowner (at 104719884831376): [0 0 0 0 0 0 0]
arr_owner (at 104719884827744): [0 0 0 0 0 0 0]

free allocated memory again ...

arr_noowner (at 104719884831376): [ -7687536     24381 -28516336     24381         0         0         0]
arr_owner (at 104719884827744): [0 0 0 0 0 0 0]

just for fun: free some python-memory ...

arr_noowner (at 104719884831376): [ -7687536     24381 -28516336     24381         0         0         0]
arr_owner (at 104719884827744): [ -7779696     24381 -28516336     24381         0         0         0]

Answer 6

Another possibility (which may require more recent versions of libraries than is available when the first answer was written -- I tested something similar with ctypes 1.1.0 and numpy 1.5.0b2) is to convert from the pointer to the array.

np.ctypeslib.as_array(
    (ctypes.c_double * array_length).from_address(ctypes.addressof(y.contents)))

This seems to still have the shared ownership semantics, so you probably need to make sure that you free the underlying buffer eventually.