What exactly is the point of memoryview in Python

Question

Checking the documentation on memoryview:

memoryview objects allow Python code to access the internal data of an object that suppor

Answer 1

memoryview objects are great when you need subsets of binary data that only need to support indexing. Instead of having to take slices (and create new, potentially large) objects to pass to another API you can just take a memoryview object.

One such API example would be the struct module. Instead of passing in a slice of the large bytes object to parse out packed C values, you pass in a memoryview of just the region you need to extract values from.

memoryview objects, in fact, support struct unpacking natively; you can target a region of the underlying bytes object with a slice, then use .cast() to 'interpret' the underlying bytes as long integers, or floating point values, or n-dimensional lists of integers. This makes for very efficient binary file format interpretations, without having to create more copies of the bytes.

Answer 2

Here is python3 code.

#!/usr/bin/env python3

import time
for n in (100000, 200000, 300000, 400000):
    data = b'x'*n
    start = time.time()
    b = data
    while b:
        b = b[1:]
    print ('bytes {:d} {:f}'.format(n,time.time()-start))

for n in (100000, 200000, 300000, 400000):
    data = b'x'*n
    start = time.time()
    b = memoryview(data)
    while b:
        b = b[1:]
    print ('memview {:d} {:f}'.format(n,time.time()-start))

Answer 3

Excellent example by Antimony. Actually, in Python3, you can replace data = 'x'*n by data = bytes(n) and put parenthesis to print statements as below:

import time
for n in (100000, 200000, 300000, 400000):
    #data = 'x'*n
    data = bytes(n)
    start = time.time()
    b = data
    while b:
        b = b[1:]
    print('bytes', n, time.time()-start)

for n in (100000, 200000, 300000, 400000):
    #data = 'x'*n
    data = bytes(n)
    start = time.time()
    b = memoryview(data)
    while b:
        b = b[1:]
    print('memoryview', n, time.time()-start)

Answer 4

One reason memoryviews are useful is because they can be sliced without copying the underlying data, unlike bytes/str.

For example, take the following toy example.

import time
for n in (100000, 200000, 300000, 400000):
    data = 'x'*n
    start = time.time()
    b = data
    while b:
        b = b[1:]
    print 'bytes', n, time.time()-start

for n in (100000, 200000, 300000, 400000):
    data = 'x'*n
    start = time.time()
    b = memoryview(data)
    while b:
        b = b[1:]
    print 'memoryview', n, time.time()-start

On my computer, I get

bytes 100000 0.200068950653
bytes 200000 0.938908100128
bytes 300000 2.30898690224
bytes 400000 4.27718806267
memoryview 100000 0.0100269317627
memoryview 200000 0.0208270549774
memoryview 300000 0.0303030014038
memoryview 400000 0.0403470993042

You can clearly see quadratic complexity of the repeated string slicing. Even with only 400000 iterations, it's already unmangeable. Meanwhile, the memoryview version has linear complexity and is lightning fast.

Edit: Note that this was done in CPython. There was a bug in Pypy up to 4.0.1 that caused memoryviews to have quadratic performance.

Answer 5

Let me make plain where lies the glitch in understanding here.

The questioner, like myself, expected to be able to create a memoryview that selects a slice of an existing array (for example a bytes or bytearray). We therefore expected something like:

desired_slice_view = memoryview(existing_array, start_index, end_index)

Alas, there is no such constructor, and the docs don't make a point of what to do instead.

The key is that you have to first make a memoryview that covers the entire existing array. From that memoryview you can create a second memoryview that covers a slice of the existing array, like this:

whole_view = memoryview(existing_array)
desired_slice_view = whole_view[10:20]

In short, the purpose of the first line is simply to provide an object whose slice implementation (dunder-getitem) returns a memoryview.

That might seem untidy, but one can rationalize it a couple of ways:

1. Our desired output is a memoryview that is a slice of something. Normally we get a sliced object from an object of that same type, by using the slice operator [10:20] on it. So there's some reason to expect that we need to get our desired_slice_view from a memoryview, and that therefore the first step is to get a memoryview of the whole underlying array.

2. The naive expection of a memoryview constructor with start and end arguments fails to consider that the slice specification really needs all the expressivity of the usual slice operator (including things like [3::2] or [:-4] etc). There is no way to just use the existing (and understood) operator in that one-liner constructor. You can't attach it to the existing_array argument, as that will make a slice of that array, instead of telling the memoryview constructor some slice parameters. And you can't use the operator itself as an argument, because it's an operator and not a value or object.

Conceivably, a memoryview constructor could take a slice object:

desired_slice_view = memoryview(existing_array, slice(1, 5, 2) )

... but that's not very satisfactory, since users would have to learn about the slice object and what its constructor's parameters mean, when they already think in terms of the slice operator's notation.