I am using numpy 1.16.2.
In brief, I am wondering how to add an object-type field to a structured array. The standard way via the recfunctions module throws an error and I suppose there is a reason for this. Therefore, I wonder whether there is anything wrong with my workaround. Furthermore, I would like to understand why this workaround is necessary and whether I need to use extra caution when accessing the newly created array.
Now here come the details:
I have a numpy structured array:
import numpy as np
a = np.zeros(3, dtype={'names':['A','B','C'], 'formats':['int','int','float']})
for i in range(len(a)):
a[i] = i
I want to add another field "test" of type object to the array a. The standard way for doing this is using numpy's recfunctions module:
import numpy.lib.recfunctions as rf
b = rf.append_fields(a, "test", [None]*len(a))
This code throws an error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-38-4a7be4f94686> in <module>
----> 1 rf.append_fields(a, "test", [None]*len(a))
D:\_Programme\Anaconda3\lib\site-packages\numpy\lib\recfunctions.py in append_fields(base, names, data, dtypes, fill_value, usemask, asrecarray)
718 if dtypes is None:
719 data = [np.array(a, copy=False, subok=True) for a in data]
--> 720 data = [a.view([(name, a.dtype)]) for (name, a) in zip(names, data)]
721 else:
722 if not isinstance(dtypes, (tuple, list)):
D:\_Programme\Anaconda3\lib\site-packages\numpy\lib\recfunctions.py in <listcomp>(.0)
718 if dtypes is None:
719 data = [np.array(a, copy=False, subok=True) for a in data]
--> 720 data = [a.view([(name, a.dtype)]) for (name, a) in zip(names, data)]
721 else:
722 if not isinstance(dtypes, (tuple, list)):
D:\_Programme\Anaconda3\lib\site-packages\numpy\core\_internal.py in _view_is_safe(oldtype, newtype)
492
493 if newtype.hasobject or oldtype.hasobject:
--> 494 raise TypeError("Cannot change data-type for object array.")
495 return
496
TypeError: Cannot change data-type for object array.
A similar error has been discussed here, though the issue is old and I do not know whether the behaviour I am observing is actually a bug. Here I am informed that views of structured arrays containing general objects are not supported.
I therefore built a workaround:
b = np.empty(len(a), dtype=a.dtype.descr+[("test", object)])
b[list(a.dtype.names)] = a
This works. Nonetheless, I have the following questions:
Questions
- Why is this workaround neccesary? Is this just a bug?
- Working with the new array
bseems to be no different from working witha. The variablec = b[["A", "test"]]is clearly a view to the data ofb. So why would they say that views on the arraybare not supported? Do I have to treatcwith extra caution?
In [161]: a = np.zeros(3, dtype={'names':['A','B','C'], 'formats':['int','int','
...: float']})
...: for i in range(len(a)):
...: a[i] = i
...:
In [162]: a
Out[162]:
array([(0, 0, 0.), (1, 1, 1.), (2, 2, 2.)],
dtype=[('A', '<i8'), ('B', '<i8'), ('C', '<f8')])
define the new dtype:
In [164]: a.dtype.descr
Out[164]: [('A', '<i8'), ('B', '<i8'), ('C', '<f8')]
In [165]: a.dtype.descr+[('test','O')]
Out[165]: [('A', '<i8'), ('B', '<i8'), ('C', '<f8'), ('test', 'O')]
In [166]: dt= a.dtype.descr+[('test','O')]
new array of right size and dtype:
In [167]: b = np.empty(a.shape, dt)
copy values from a to b by field name:
In [168]: for name in a.dtype.names:
...: b[name] = a[name]
...:
In [169]: b
Out[169]:
array([(0, 0, 0., None), (1, 1, 1., None), (2, 2, 2., None)],
dtype=[('A', '<i8'), ('B', '<i8'), ('C', '<f8'), ('test', 'O')])
Many of the rf functions do this field by field copy:
rf.recursive_fill_fields(a,b)
rf.append_fields uses this after it initializes it's output array.
In earlier versions a multifield index produced a copy, so expressions like b[list(a.dtype.names)] = a would not work.
I don't know if it's worth trying to figure out what rf.append_fields is doing. Those functions are somewhat old, and not heavily used (note the special import). So it's entirely likely that they have bugs, or edge cases , that don't work. The functions that I've examined function much as I demonstrated - make a new dtype, and result array, and copy data by field name.
In recent releases there have been changes in how multiple fields are accessed. There are some new functions in recfunctions to facilitate working with structured arrays, such as repack_fields.
https://docs.scipy.org/doc/numpy/user/basics.rec.html#accessing-multiple-fields
I don't know if any of that applies to the append_fields problem. I see there's also a section about structured arrays with objects, but I haven't studied that:
https://docs.scipy.org/doc/numpy/user/basics.rec.html#viewing-structured-arrays-containing-objects
In order to prevent clobbering object pointers in fields of numpy.object type, numpy currently does not allow views of structured arrays containing objects.
This line apparently refers to the use of view method. Views created by field indexing, whether single name or multifield lists, are not affected.
The error in append_fields comes from this operation:
In [183]: data = np.array([None,None,None])
In [184]: data
Out[184]: array([None, None, None], dtype=object)
In [185]: data.view([('test',object)])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-185-c46c4464b53c> in <module>
----> 1 data.view([('test',object)])
/usr/local/lib/python3.6/dist-packages/numpy/core/_internal.py in _view_is_safe(oldtype, newtype)
492
493 if newtype.hasobject or oldtype.hasobject:
--> 494 raise TypeError("Cannot change data-type for object array.")
495 return
496
TypeError: Cannot change data-type for object array.
There's no problem creating an compound dtype with object dtypes:
In [186]: np.array([None,None,None], dtype=[('test',object)])
Out[186]: array([(None,), (None,), (None,)], dtype=[('test', 'O')])
But I don't see any recfunctions that are capable of joining a and data.
view can be used to change the field names of a:
In [219]: a.view([('AA',int),('BB',int),('cc',float)])
Out[219]:
array([(0, 0, 0.), (1, 1, 1.), (2, 2, 2.)],
dtype=[('AA', '<i8'), ('BB', '<i8'), ('cc', '<f8')])
but trying to do so for b fails for the same reason:
In [220]: b.view([('AA',int),('BB',int),('cc',float),('d',object)])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-220-ab0a6e4dd57f> in <module>
----> 1 b.view([('AA',int),('BB',int),('cc',float),('d',object)])
/usr/local/lib/python3.6/dist-packages/numpy/core/_internal.py in _view_is_safe(oldtype, newtype)
492
493 if newtype.hasobject or oldtype.hasobject:
--> 494 raise TypeError("Cannot change data-type for object array.")
495 return
496
TypeError: Cannot change data-type for object array.
I start with a object dtype array, and try to view with i8 (same size dtype), I get this same error. So the restriction on view of a object dtype isn't limited to structured arrays. The need for such a restriction in the case of object pointer to i8 makes sense. The need for such a restriction in the case of embedding the object pointer in a compound dtype might not be so compelling. It might even be overkill, or just a case of simply playing it safe and simple.
In [267]: x.dtype
Out[267]: dtype('O')
In [268]: x.shape
Out[268]: (3,)
In [269]: x.dtype.itemsize
Out[269]: 8
In [270]: x.view('i8')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-270-30c78b13cd10> in <module>
----> 1 x.view('i8')
/usr/local/lib/python3.6/dist-packages/numpy/core/_internal.py in _view_is_safe(oldtype, newtype)
492
493 if newtype.hasobject or oldtype.hasobject:
--> 494 raise TypeError("Cannot change data-type for object array.")
495 return
496
TypeError: Cannot change data-type for object array.
Note that the test in line 493 checks the hasobject property of both the new and old dtypes. A more nuanced test might check if both hasobject, but I suspect the logic could get quite complex. Sometimes a simple prohibition is safer (and easier) a complex set of tests.
In further testing
In [283]: rf.structured_to_unstructured(a)
Out[283]:
array([[ 3., 3., 0.],
[12., 10., 1.],
[ 2., 2., 2.]])
but trying to do the same on b, or even a subset of its fields produces the familiar error:
rf.structured_to_unstructured(b)
rf.structured_to_unstructured(b[['A','B','C']])
I have to first use repack to make a object-less copy:
rf.structured_to_unstructured(rf.repack_fields(b[['A','B','C']]))
来源：https://stackoverflow.com/questions/55718258/add-and-access-object-type-field-of-a-numpy-structured-array