scipy and preserving mat file (.mat matlab data file) structure

Question

After referring to scipy and numpy docs for a day and a half, I tried doing this -

dt = {\'names\':[u\'OSversInt\',u\'Desc\',u\'OSversStr\',\\
... u\'OSname\',u         


        

Answer 1

In Octave I created a cell with a structure object:

octave:14> y={x}
y = 
{
  [1,1] =

    scalar structure containing the fields:

      OSversInt =  15
      Desc = 
      OSverStr = 5.0.1
      OSname = Android

}
octave:15> save stack32723802.mat -V7 y

In numpy I load it as:

In [376]: L=loadmat('stack32723802.mat')
In [377]: L['y']
Out[377]: 
array([[ array([[([[15.0]], [], ['5.0.1'], ['Android'])]], 
      dtype=[('OSversInt', 'O'), ('Desc', 'O'), ('OSverStr', 'O'), ('OSname', 'O')])]], 
      dtype=object)

That's a 2d object array (1,1), with one item, which is also 2d (1,1), with the compound dtype.

In [390]: y=L['y']
In [391]: y[0,0]
Out[391]: 
array([[([[15.0]], [], ['5.0.1'], ['Android'])]], 
      dtype=[('OSversInt', 'O'), ('Desc', 'O'), ('OSverStr', 'O'), ('OSname', 'O')])

In [392]: y[0,0][0,0]
Out[392]: ([[15.0]], [], ['5.0.1'], ['Android'])

In [394]: y[0,0]['OSversInt']
Out[394]: array([[array([[ 15.]])]], dtype=object)

In [395]: y[0,0]['OSname']
Out[395]: 
array([[array(['Android'], 
      dtype='<U7')]], dtype=object)

The 4d 'actual outcome' is the result of np.array producing the highest dimensional array it can.

First create the inner structured array:

In [405]: dt=y.item().dtype
In [406]: item=([[15.0]], [], ['5.0.1'], ['Android'])
In [407]: array1 = np.array([[item]], dtype=dt)
In [408]: array1
Out[408]: 
array([[([[15.0]], [], ['5.0.1'], ['Android'])]], 
      dtype=[('OSversInt', 'O'), ('Desc', 'O'), ('OSverStr', 'O'), ('OSname', 'O')])

If I just wrap it in a 2d object array, I get a 4d array:

In [409]: np.array([[array1]], dtype=object)
Out[409]: array([[[[([[15.0]], [], ['5.0.1'], ['Android'])]]]], dtype=object)

But if instead I create an empty 2d object array, and insert this inner array, I get something that matches the loadmat result:

In [410]: z=np.empty((1,1),dtype=object)
In [411]: z[0,0]=np.array([[item]], dtype=dt)
In [412]: z
Out[412]: 
array([[ array([[([[15.0]], [], ['5.0.1'], ['Android'])]], 
      dtype=[('OSversInt', 'O'), ('Desc', 'O'), ('OSverStr', 'O'), ('OSname', 'O')])]], dtype=object)

Answer 2

Based on hpaulj's comment it appears you may be looking for

aa = np.empty((1,1), dtype='O')
aa[0,0] = np.array([[ ([[15]], [], [u'5.0.1'], [u'Android'], [u'main'], [u'MSM8960'])]], 
          dtype=[('OSversInt', 'O'), ('Desc', 'O'), ('OSversStr', 'O'),
                 ('OSname', 'O'), ('platform', 'O'), ('Board', 'O')])

which yields

In [39]: aa
Out[39]: 
array([[ array([[([[15]], [], [u'5.0.1'], [u'Android'], [u'main'], [u'MSM8960'])]], 
      dtype=[('OSversInt', 'O'), ('Desc', 'O'), ('OSversStr', 'O'), ('OSname', 'O'), ('platform', 'O'), ('Board', 'O')])]], dtype=object)

---

When you want to place an arbitrary Python object (such as a NumPy array or a list) inside of a NumPy array, the dtype must be object. In this case construction with np.array fails because this function interprets inner sequences (other than tuples) as values to be recursed upon instead of as atomic elements.

So the trick to creating these nested object arrays is to create the outer object array first:

aa = np.empty((1,1), dtype='O')

and then at assign the desired value to the cells of the array:

aa[0,0] = ...

Note that nested NumPy arrays of dtype object do not allow you to take advantage of NumPy's fast (mainly numeric) functions. They have essentially the same memory footprint as the Python objects they contain, and performance-wise they are typically no better than plain Python lists of lists.