Using scipy.stats.stats in django after deployment

Question

I am in the process of creating an django-powered (1.3) interface to a package that relies heavily in scipy.stats.stats (scipy version 0.9.0), called ovl . During early development phases, using djangos own development-server, this was no problem. After deployment using apache debian/2.2.9 and mod_wsgi 3.3, this causes a serious problem.

Whatever view I'm trying to load in the browser, it starts loading, and keeps doing that for a good 5 minutes (until time-out) and a 500 page appears. Just importing scipy works but, does not make scipy.stats.stats or even scipy.stats available. This is no surprise; in the documentation in scipy's init.py it is stated that the subpackage stats needs to be imported explicitly. However, the same is said about the subpackage cluster, which imports in django (from the web and in the django-shell) without any problem and indeed shows up in dir(scipy), which it doesn't in an ipython(0.10.2)-session, where it just doesn't show up, like I kinda expected.

On the command dir(scipy); it returns different results when coming from the web (a list of 568 strings, including the subpackage cluster) in the normal ipython shell (564 strings, no subpackage cluster) and surprise, surprise, in the django shell. In the django shell scipy has 570 attributes, including both cluster and stats packages.

Another thing is, if I keep importing the ovl-package, while keeping the scipy.stats imports at a bit of a distance (not in one of the files of the app itself), sometimes I get a ViewDoesNotExist error stating that there is no method index in the views module while there clearly is one. Which reminds me of this.

So now I'm thinking of these rather ugly solutions:

• Editing scipy's init to import the stats package so it appears 'normally' in dir(scipy) and is accessible through scipy.stats and I can use the old code.
• Snatching scipy's stat subpackage and making a regular package out of it (perhaps using a symlink)

I'm reluctant in applying these solutions, however. The fact cluster shows up in scipy in a django environment worries me a bit. I thought maybe this has something to do with being www-data user when logging in from the web, but I don't know how to check for that.

Did anyone else encounter this? Parts of this? Or otherwise helpful thoughts?

Oh and another django deployment does work.

Answer 1

mod_python used to try using multiple Python interpreters in the same process. mod_wsgi might be doing the same. While this frequently works okay, some extension modules do not support this. scipy.stats is probably importing such an extension module. We have had similar reports on the scipy mailing list concerning scipy.stats under mod_python. Check the mod_wsgi documentation to see if you can configure it such that it does not use multiple interpreters in the same process, or find a different deployment strategy that uses one interpreter per process for the app.

Answer 2

I also encounter this problem when I use scipy.stats in my django application. In django manage.py runsever environment, my app is executed correctly without any problem. But when I deploy the app to apache server with mod_python, I can't enter my app, and the browser keep loading until the time out. After I remove all imported scipy.stats statements in my app, the problem is solved, and I can execute my app in the apache server.