问题
I'm creating a dictionary d of one million of items which are tuples, and ideally I'd like to access them with:
d[1634].id # or d[1634]['id']
d[1634].name # or d[1634]['name']
d[1634].isvalid # or d[1634]['isvalid']
rather than d[1634][0], d[1634][1], d[1634][2] which is less explicit.
According to my test:
import os, psutil, time, collections, typing
Tri = collections.namedtuple('Tri', 'id,name,isvalid')
Tri2 = typing.NamedTuple("Tri2", [('id', int), ('name', str), ('isvalid', bool)])
t0 = time.time()
# uncomment only one of these 4 next lines:
d = {i: (i+1, 'hello', True) for i in range(1000000)} # tuple
# d = {i: {'id': i+1, 'name': 'hello', 'isvalid': True} for i in range(1000000)} # dict
# d = {i: Tri(id=i+1, name='hello', isvalid=True) for i in range(1000000)} # namedtuple
# d = {i: Tri2(id=i+1, name='hello', isvalid=True) for i in range(1000000)} # NamedTuple
print('%.3f s %.1f MB' % (time.time()-t0, psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2))
"""
tuple: 0.257 s 193.3 MB
dict: 0.329 s 363.6 MB
namedtuple: 1.253 s 193.3 MB (collections)
NamedTuple: 1.250 s 193.5 MB (typing)
"""
- using a
dictdoubles the RAM usage, compared to atuple - using a
namedtupleorNamedTuplemultiplies by 5 the time spent, compared to atuple!
Question: is there a tuple-like data structure in Python 3 which allows to access the data with x.id, x.name, etc. and also is RAM and CPU efficient?
Notes:
in my real use case, the
tupleis something like a C-struct of type(uint64, uint64, bool).I've also tried with:
slots(to avoid the interal object's__dict__, see Usage of __slots__?)dataclass:@dataclasses.dataclass class Tri3: id: int ...ctypes.Structure:class Tri7(ctypes.Structure): _fields_ = [("id", ctypes.c_int), ...]
but it was not better (all of them ~ 1.2 sec.), nothing close to a genuine
tuplein terms of performanceHere are other options: C-like structures in Python
回答1:
Cython's cdef-classes might be what you want: They use less memory than the pure Python classes, even if it comes at costs of more overhead when accessing members (because fields are stored as C-values and not Python-objects).
For example:
%%cython
cdef class CTuple:
cdef public unsigned long long int id
cdef public str name
cdef public bint isvalid
def __init__(self, id, name, isvalid):
self.id = id
self.name = name
self.isvalid = isvalid
which can be used as wished:
ob=CTuple(1,"mmm",3)
ob.id, ob.name, ob.isvalid # prints (2, "mmm", 3)
Timings/memory consumption:
First, the baseline on my machine:
0.258 s 252.4 MB # tuples
0.343 s 417.5 MB # dict
1.181 s 264.0 MB # namedtuple collections
with CTuple we get:
0.306 s 191.0 MB
which is almost as fast and needs considerable less memory.
If the C-type of members isn't clear at compile time, one could use simple python-objects:
%%cython
cdef class PTuple:
cdef public object id
cdef public object name
cdef public object isvalid
def __init__(self, id, name, isvalid):
self.id = id
self.name = name
self.isvalid = isvalid
The timings are a little bit surprising:
0.648 s 249.8 MB
I didn't expect it to be so much slower than the CTuple-version, but at least it is twice as fast as named tuples.
One disadvantage of this approach is that it needs compilation. Cython however offers cython.inline which can be used to compile Cython-code created on-the-fly.
I've released cynamedtuple which can be installed via pip install cynamedtuple, and is based on the prototype bellow:
import cython
# for generation of cython code:
tab = " "
def create_members_definition(name_to_ctype):
members = []
for my_name, my_ctype in name_to_ctype.items():
members.append(tab+"cdef public "+my_ctype+" "+my_name)
return members
def create_signature(names):
return tab + "def __init__(self,"+", ".join(names)+"):"
def create_initialization(names):
inits = [tab+tab+"self."+x+" = "+x for x in names]
return inits
def create_cdef_class_code(classname, names):
code_lines = ["cdef class " + classname + ":"]
code_lines.extend(create_members_definition(names))
code_lines.append(create_signature(names.keys()))
code_lines.extend(create_initialization(names.keys()))
return "\n".join(code_lines)+"\n"
# utilize cython.inline to generate and load pyx-module:
def create_cnamedtuple_class(classname, names):
code = create_cdef_class_code(classname, names)
code = code + "GenericClass = " + classname +"\n"
ret = cython.inline(code)
return ret["GenericClass"]
Which can be used as follows, to dynamically define CTuple from above:
CTuple = create_cnamedtuple_class("CTuple",
{"id":"unsigned long long int",
"name":"str",
"isvalid":"bint"})
ob = CTuple(1,"mmm",3)
...
Another alternative could be to use jit-compilation and Numba's jitted-classes which offer this possibility. They however seem to be much slower:
from numba import jitclass, types
spec = [
('id', types.uint64),
('name', types.string),
('isvalid', types.uint8),
]
@jitclass(spec)
class NBTuple(object):
def __init__(self, id, name, isvalid):
self.id = id
self.name = name
self.isvalid = isvalid
and the results are:
20.622 s 394.0 MB
so numba jitted classes are not (yet?) a good choice.
回答2:
You can try to reverse it (store as struct of arrays) and access the values as x['id'][1634]. In other words, x is a dictionary with three keys and value for each key is a list. It will be space efficient.
Or you can use pandas dataframes. Dataframes are stored in a matrix form where the rows have numeric IDs and columns have labels (strings like 'name' etc.). For a dataframe df, df.iloc[i] points to the $i^th$ row and you can access the name in that row by df.iloc[i].name or df.iloc[i]['name']
来源：https://stackoverflow.com/questions/65159938/dictionary-of-named-tuples-in-python-and-speed-ram-performance