Speeding Up Python

Question

This is really two questions, but they are so similar, and to keep it simple, I figured I\'d just roll them together:

• Firstly: Given an est

Answer 1

Run your app through the Python profiler. Find a serious bottleneck. Rewrite that bottleneck in C. Repeat.

Answer 2

It's often possible to achieve near-C speeds (close enough for any project using Python in the first place!) by replacing explicit algorithms written out longhand in Python with an implicit algorithm using a built-in Python call. This works because most Python built-ins are written in C anyway. Well, in CPython of course ;-) https://www.python.org/doc/essays/list2str/

Answer 3

The canonical reference to how to improve Python code is here: PerformanceTips. I'd recommend against optimizing in C unless you really need to though. For most applications, you can get the performance you need by following the rules posted in that link.

Answer 4

I'm surprised no one mentioned ShedSkin: http://code.google.com/p/shedskin/, it automagically converts your python program to C++ and in some benchmarks yields better improvements than psyco in speed.

Plus anecdotal stories on the simplicity: http://pyinsci.blogspot.com/2006/12/trying-out-latest-release-of-shedskin.html

There are limitations though, please see: http://tinyurl.com/shedskin-limitations

Answer 5

Rather than just punting to C, I'd suggest:

Make your code count. Do more with fewer executions of lines:

• Change the algorithm to a faster one. It doesn't need to be fancy to be faster in many cases.
• Use python primitives that happens to be written in C. Some things will force an interpreter dispatch where some wont. The latter is preferable
• Beware of code that first constructs a big data structure followed by its consumation. Think the difference between range and xrange. In general it is often worth thinking about memory usage of the program. Using generators can sometimes bring O(n) memory use down to O(1).
• Python is generally non-optimizing. Hoist invariant code out of loops, eliminate common subexpressions where possible in tight loops.
• If something is expensive, then precompute or memoize it. Regular expressions can be compiled for instance.
• Need to crunch numbers? You might want to check numpy out.
• Many python programs are slow because they are bound by disk I/O or database access. Make sure you have something worthwhile to do while you wait on the data to arrive rather than just blocking. A weapon could be something like the Twisted framework.
• Note that many crucial data-processing libraries have C-versions, be it XML, JSON or whatnot. They are often considerably faster than the Python interpreter.

If all of the above fails for profiled and measured code, then begin thinking about the C-rewrite path.

Answer 6

This is the procedure that I try to follow:

• import psyco; psyco.full()
• If it's not fast enough, run the code through a profiler, see where the bottlenecks are. (DISABLE psyco for this step!)
• Try to do things such as other people have mentioned to get the code at those bottlenecks as fast as possible.
  • Stuff like [str(x) for x in l] or [x.strip() for x in l] is much, much slower than map(str, x) or map(str.strip, x).
• After this, if I still need more speed, it's actually really easy to get PyRex up and running. I first copy a section of python code, put it directly in the pyrex code, and see what happens. Then I twiddle with it until it gets faster and faster.