Efficient way for appending numpy array

Question

I will keep it simple.I have a loop that appends new row to a numpy array...what is the efficient way to do this.

n=np.zeros([1,2])
for x in [[2,3],[4,5],[7,         


        

Answer 1

You might want to try:

import numpy as np

n = np.reshape([], (0, 2))
for x in [[2,3],[4,5],[7,6]]:
      n = np.append(n, [x], axis=0)

Instead of np.append you can also use n = np.vstack([n,x]). I also agree with @Bi Rico that I also would use a list, if n does not need to accessed within the loop.

Answer 2

A bit of technical explanation for the "why lists" part.

Internally, the problem for a list of unknown length is that it needs to fit in memory somehow regardless of its length. There are essentially two different possibilities:

1. Use a data structure (linked list, some tree structure, etc.) which makes it possible to allocate memory separately for each new element in a list.

2. Store the data in a contiguous memory area. This area has to be allocated when the list is created, and it has to be larger than what we initially need. If we get more stuff into the list, we need to try to allocate more memory, preferably at the same location. If we cannot do it at the same location, we need to allocate a bigger block and move all data.

The first approach enables all sorts of fancy insertion and deletion options, sorting, etc. However, it is slower in sequential reading and allocates more memory. Python actually uses the method #2, the lists are stored as "dynamic arrays". For more information on this, please see:

Size of list in memory

What this means is that lists are designed to be very efficient with the use of append. There is very little you can do to speed things up if you do not know the size of the list beforehand.

---

If you know even the maximum size of the list beforehand, you are probably best off allocating a numpy.array using numpy.empty (not numpy.zeros) with the maximum size and then use ndarray.resize to shrink the array once you have filled in all data.

For some reason numpy.array(l) where l is a list is often slow with large lists, whereas copying even large arrays is quite fast (I just tried to create a copy of a 100 000 000 element array; it took less than 0.5 seconds).

This discussion has more benchmarking on different options:

Fastest way to grow a numpy numeric array

I have not benchmarked the numpy.empty + ndarray.resize combo, but both should be rather microsecond than millisecond operations.

Answer 3

There are three ways to do this, if you already have everything in a list:

data = [[2, 3], [4, 5], [7, 6]]
n = np.array(data)

If you know how big the final array will be:

exp = np.array([2, 3])    

n = np.empty((3, 2))
for i in range(3):
    n[i, :] = i ** exp

If you don't know how big the final array will be:

exp = np.array([2, 3])

n = []
i = np.random.random()
while i < .9:
    n.append(i ** exp)
    i = np.random.random()
n = np.array(n)

Just or the record you can start with n = np.empty((0, 2)) but I would not suggest appending to that array in a loop.