Speeding Up Python

Question

This is really two questions, but they are so similar, and to keep it simple, I figured I\'d just roll them together:

• Firstly: Given an est

Answer 1

If using psyco, I'd recommend psyco.profile() instead of psyco.full(). For a larger project it will be smarter about the functions that got optimized and use a ton less memory.

I would also recommend looking at iterators and generators. If your application is using large data sets this will save you many copies of containers.

Answer 2

First thing that comes to mind: psyco. It runs only on x86, for the time being.

Then, constant binding. That is, make all global references (and global.attr, global.attr.attr…) be local names inside of functions and methods. This isn't always successful, but in general it works. It can be done by hand, but obviously is tedious.

You said apart from in-code optimization, so I won't delve into this, but keep your mind open for typical mistakes (for i in range(10000000) comes to mind) that people do.

Answer 3

The usual suspects -- profile it, find the most expensive line, figure out what it's doing, fix it. If you haven't done much profiling before, there could be some big fat quadratic loops or string duplication hiding behind otherwise innocuous-looking expressions.

In Python, two of the most common causes I've found for non-obvious slowdown are string concatenation and generators. Since Python's strings are immutable, doing something like this:

result = u""
for item in my_list:
    result += unicode (item)

will copy the entire string twice per iteration. This has been well-covered, and the solution is to use "".join:

result = "".join (unicode (item) for item in my_list)

Generators are another culprit. They're very easy to use and can simplify some tasks enormously, but a poorly-applied generator will be much slower than simply appending items to a list and returning the list.

Finally, don't be afraid to rewrite bits in C! Python, as a dynamic high-level language, is simply not capable of matching C's speed. If there's one function that you can't optimize any more in Python, consider extracting it to an extension module.

My favorite technique for this is to maintain both Python and C versions of a module. The Python version is written to be as clear and obvious as possible -- any bugs should be easy to diagnose and fix. Write your tests against this module. Then write the C version, and test it. Its behavior should in all cases equal that of the Python implementation -- if they differ, it should be very easy to figure out which is wrong and correct the problem.

Answer 4

People have given some good advice, but you have to be aware that when high performance is needed, the python model is: punt to c. Efforts like psyco may in the future help a bit, but python just isn't a fast language, and it isn't designed to be. Very few languages have the ability to do the dynamic stuff really well and still generate very fast code; at least for the forseeable future (and some of the design works against fast compilation) that will be the case.

So, if you really find yourself in this bind, your best bet will be to isolate the parts of your system that are unacceptable slow in (good) python, and design around the idea that you'll rewrite those bits in C. Sorry. Good design can help make this less painful. Prototype it in python first though, then you've easily got a sanity check on your c, as well.

This works well enough for things like numpy, after all. I can't emphasize enough how much good design will help you though. If you just iteratively poke at your python bits and replace the slowest ones with C, you may end up with a big mess. Think about exactly where the C bits are needed, and how they can be minimized and encapsulated sensibly.

Answer 5

Regarding "Secondly: When writing a program from scratch in python, what are some good ways to greatly improve performance?"

Remember the Jackson rules of optimization:

• Rule 1: Don't do it.
• Rule 2 (for experts only): Don't do it yet.

And the Knuth rule:

• "Premature optimization is the root of all evil."

The more useful rules are in the General Rules for Optimization.

1. Don't optimize as you go. First get it right. Then get it fast. Optimizing a wrong program is still wrong.

2. Remember the 80/20 rule.

3. Always run "before" and "after" benchmarks. Otherwise, you won't know if you've found the 80%.

4. Use the right algorithms and data structures. This rule should be first. Nothing matters as much as algorithm and data structure.

Bottom Line

You can't prevent or avoid the "optimize this program" effort. It's part of the job. You have to plan for it and do it carefully, just like the design, code and test activities.

Answer 6

Besides the (great) psyco and the (nice) shedskin, I'd recommend trying cython a great fork of pyrex.

Or, if you are not in a hurry, I recommend to just wait. Newer python virtual machines are coming, and unladen-swallow will find its way into the mainstream.