How can this Python Scrabble word finder be made faster?

Question

I have no real need to improve it, it\'s just for fun. Right now it\'s taking about a second on a list of about 200K words.

I\'ve tried to optimize it as much as I know

Answer 1

import trie


def walk_trie(trie_node, rack, path=""):
    if trie_node.value is None:
        yield path
    for i in xrange(len(rack)):
        sub_rack = rack[:i] + rack[i+1:]
        if trie_node.nodes.has_key(rack[i]):
            for word in walk_trie(trie_node.nodes[rack[i]], sub_rack, path+rack[i]):
                yield word


if __name__ == "__main__":
    print "Generating trie... "

    # You might choose to skip words starting with a capital
    # rather than lower-casing and searching everything. Capitalised
    # words are probably pronouns which aren't allowed in Scrabble

    # I've skipped words shorter than 3 characters.
    all_words = ((line.strip().lower(), None) for line in open("/usr/share/dict/words") if len(line.strip()) >= 3)
    word_trie = trie.Trie(mapping=all_words)
    print "Walking Trie... "
    print list(walk_trie(word_trie.root, "abcdefg"))

Generating the trie takes a little while, but once generated getting the list of words should be much faster than looping over a list.

If anyone knows a way to serialise a trie that would be a great addition.

Just to demonstrate that generating the trie is what takes the time...

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    98333    5.344    0.000    8.694    0.000 trie.py:87(__setitem__)
   832722    1.849    0.000    1.849    0.000 trie.py:10(__init__)
   832721    1.501    0.000    1.501    0.000 {method 'setdefault' of 'dict' objects}
    98334    1.005    0.000    1.730    0.000 scrabble.py:16()
        1    0.491    0.491   10.915   10.915 trie.py:82(extend)
   196902    0.366    0.000    0.366    0.000 {method 'strip' of 'str' objects}
    98333    0.183    0.000    0.183    0.000 {method 'lower' of 'str' objects}
    98707    0.177    0.000    0.177    0.000 {len}
   285/33    0.003    0.000    0.004    0.000 scrabble.py:4(walk_trie)
      545    0.001    0.000    0.001    0.000 {method 'has_key' of 'dict' objects}
        1    0.001    0.001   10.921   10.921 {execfile}
        1    0.001    0.001   10.920   10.920 scrabble.py:1()
        1    0.000    0.000    0.000    0.000 trie.py:1()
        1    0.000    0.000    0.000    0.000 {open}
        1    0.000    0.000    0.000    0.000 trie.py:5(Node)
        1    0.000    0.000   10.915   10.915 trie.py:72(__init__)
        1    0.000    0.000    0.000    0.000 trie.py:33(Trie)
        1    0.000    0.000   10.921   10.921 :1()
        1    0.000    0.000    0.000    0.000 {method 'split' of 'str' objects}
        1    0.000    0.000    0.000    0.000 trie.py:1(NeedMore)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}