Django (?) really slow with large datasets after doing some python profiling

Question

I was comparing an old PHP script of mine versus the newer, fancier Django version and the PHP one, with full spitting out of HTML and all was functioning faster. MUCH faster to

Answer 1

"tokenize.py comes out on top, which can make some sense as I am doing a lot of number formatting. "

Makes no sense at all.

See http://docs.python.org/library/tokenize.html.

The tokenize module provides a lexical scanner for Python source code, implemented in Python

Tokenize coming out on top means that you have dynamic code parsing going on.

AFAIK (doing a search on the Django repository) Django does not use tokenize. So that leaves your program doing some kind of dynamic code instantiation. Or, you're only profiling the first time your program is loaded, parsed and run, leading to false assumptions about where the time is going.

You should not ever do calculation in template tags -- it's slow. It involves a complex meta-evaluation of the template tag. You should do all calculations in the view in simple, low-overhead Python. Use the templates for presentation only.

Also, if you're constantly doing queries, filters, sums, and what-not, you have a data warehouse. Get a book on data warehouse design, and follow the data warehouse design patterns.

You must have a central fact table, surrounded by dimension tables. This is very, very efficient.

Sums, group bys, etc., are can be done as defaultdict operations in Python. Bulk fetch all the rows, building the dictionary with the desired results. If this is too slow, then you have to use data warehousing techniques of saving persistent sums and groups separate from your fine-grained facts. Often this involves stepping outside the Django ORM and using RDBMS features like views or tables of derived data.