I got a text file containing all my data
data = \'B:/tempfiles/bla.dat\'
from the text file I\'m listing the column header and their types with
The behaviour is documented, the NameValidator class in lib/_iotools.py
which parses the names passed in to genfromtxt
:
class NameValidator(object):
"""
Object to validate a list of strings to use as field names.
The strings are stripped of any non alphanumeric character, and spaces
are replaced by '_'. During instantiation, the user can define a list
of names to exclude, as well as a list of invalid characters. Names in
the exclusion list are appended a '_' character.
Once an instance has been created, it can be called with a list of
names, and a list of valid names will be created. The `__call__`
method accepts an optional keyword "default" that sets the default name
in case of ambiguity. By default this is 'f', so that names will
default to `f0`, `f1`, etc.
The relevant line in your case being The strings are stripped of any non alphanumeric character
You can see the behaviour by calling the NameValidator.validate
on a list with other non alphanumeric characters in the names:
In [17]: from numpy.lib._iotools import NameValidator
In [18]: l = ["foo(1)","bar!!!","foo bar??"]
In [19]: NameValidator().validate(l)
Out[19]: ('foo1', 'bar', 'foo_bar')
And the same using genfromtxt:
In [24]: datafile = np.genfromtxt("foo.txt", dtype=[('foo!! bar??', '<f8'), ('foo bar bar$', '<f8')], delimiter=",",defaultfmt="%")
In [25]: datafile.dtype
Out[25]: dtype([('foo_bar', '<f8'), ('foo_bar_bar', '<f8')])
When you have problems with genfromtxt
the first thing you should do is print the shape
and dtype
.
Why do you have to use ()
in col_headers = [('VW_3_Avg','<f8'),('Lvl_Max(1)','<f8')]
?
Is it because the file has those names in the header?
If you are giving your own dtype
and using skip_header
it doesn't matter what's on the file. It's the field names in the dtype
that count, not the ones on the file.
We could dig in to the dtype
documentation and find just what characters are allowed. Field names that would work as Python variable names certainly will work. I'm not surprised the ()
would be disallowed or have problems, though I haven't tested that.
Actually 'Lvl_Max(1)'
is acceptable as a dtype field name:
In [235]: col_headers = [('VW_3_Avg','<f8'),('Lvl_Max(1)','<f8')]
In [236]: A=np.zeros((3,),dtype=col_headers)
In [237]: A
Out[237]:
array([(0.0, 0.0), (0.0, 0.0), (0.0, 0.0)],
dtype=[('VW_3_Avg', '<f8'), ('Lvl_Max(1)', '<f8')])
In [238]: A['Lvl_Max(1)']
Out[238]: array([ 0., 0., 0.])
What you should have done, right from the start, is show us datafile.shape
and datafile.dtype
. 90% of these genfromtxt
problems stem from a misunderstanding of the function returns.
Let's try a simple fileread with this dtype:
In [239]: txt=b"""1 2
.....: 3 4
.....: 5 6
.....: """
In [240]: np.genfromtxt(txt.splitlines(),dtype=col_headers)
Out[240]:
array([(1.0, 2.0), (3.0, 4.0), (5.0, 6.0)],
dtype=[('VW_3_Avg', '<f8'), ('Lvl_Max1', '<f8')])
Look at the dtype
. genfromtxt
has stripped off the '(1)'
. Looks like genfromtxt
'sanitizes' the field names, no doubt because names on text file could have all kinds of funny stuff.
From the genfromtxt
docs:
Numpy arrays with a structured dtype can also be viewed as recarray, where a field can be accessed as if it were an attribute. For that reason, we may need to make sure that the field name doesn’t contain any space or invalid character, or that it does not correspond to the name of a standard attribute (like size or shape), which would confuse the interpreter.
genfromtxt
takes a deletechars
parameter that should let you control which characters are deleted from the field names. But it's application is inconsistent.
In [282]: np.genfromtxt(txt.splitlines(),names=np.dtype(col_headers).names,deletechars=set(b' '),dtype=None)
Out[282]:
array([(1, 2), (3, 4), (5, 6)],
dtype=[('VW_3_Avg', '<i4'), ('Lvl_Max(1)', '<i4')])
In [283]: np.genfromtxt(txt.splitlines(),names=np.dtype(col_headers).names,deletechars=set(b' '))
Out[283]:
array([(1.0, 2.0), (3.0, 4.0), (5.0, 6.0)],
dtype=[('VW_3_Avg', '<f8'), ('Lvl_Max1', '<f8')])
dtype=None
is required for this to work.
The default set is large:
defaultdeletechars = set("""~!@#$%^&*()-=+~\|]}[{';: /?.>,<""")
The problem is that deletechars
is passed to the validator
:
validate_names = NameValidator(...
deletechars=deletechars,...)
which is used to clean names from the header and the names
parameter. But then the names (and dtype) are passed through
dtype = easy_dtype(dtype, defaultfmt=defaultfmt, names=names)
without the deletechars
parameter. This issue was addressed about a year ago, https://github.com/numpy/numpy/pull/4649, so may be fixed in new(est) versions.