Obtaining pointer to python memoryview on bytes object

Question

I have a python memoryview pointing to a bytes object on which I would like to perform some processing in cython.

My problem is:

• because the bytes object is not writable, cython does not allow constructing a typed (cython) memoryview from it
• I cannot use pointers either because I cannot get a pointer to the memoryview start

Example:

In python:

array = memoryview(b'abcdef')[3:]

In cython:

• cdef char * my_ptr = &array[0] fails to compile with the message: Cannot take address of Python variable
• cdef char[:] my_view = array fails at runtime with the message: BufferError: memoryview: underlying buffer is not writable

How does one solve this?

Answer 1

Ok, after digging through the python api I found a solution to get a pointer to the bytes object's buffer in a memoryview (here called bytes_view = memoryview(bytes())). Maybe this helps somebody else:

from cpython.buffer cimport PyObject_GetBuffer, PyBuffer_Release, PyBUF_ANY_CONTIGUOUS, PyBUF_SIMPLE


cdef Py_buffer buffer
cdef char * my_ptr

PyObject_GetBuffer(bytes, &buffer, PyBUF_SIMPLE | PyBUF_ANY_CONTIGUOUS)
try:
    my_ptr = <char *>buffer.buf
    # use my_ptr
finally:
    PyBuffer_Release(&buffer)

Answer 2

Using a bytearray (as per @CheeseLover's answer) is probably the right way of doing things. My advice would be to work entirely in bytearrays thereby avoiding temporary conversions. However:

char* can be directly created from a Python string (or bytes) - see the end of the linked section:

cdef char * my_ptr = array
# you can then convert to a memoryview as normal in Cython
cdef char[:] mview = <char[:len(array)]>my_ptr

A couple of warnings:

1. Remember that bytes is not mutable and if you attempt to modify that memoryview is likely to cause issues
2. my_ptr (and thus mview) are only valid so long as array is valid, so be sure to keep a reference to array for as long as you need access ti the data,

Answer 3

You can use bytearray to create a mutable memoryview. Please note that this won't change the string, only the bytearray

data = bytearray('python')
view = memoryview(data)
view[0] = 'c'
print data
# cython

Answer 4

If you don't want cython memoryview to fail with 'underlying buffer is not writable' you simply should not ask for a writable buffer. Once you're in C domain you can summarily deal with that writability. So this works:

cdef const unsigned char[:] my_view = array
cdef char* my_ptr = <char*>&my_view[0]