Testing for mysterious load errors in python/django

Question

This is related to this Configure Apache to recover from mod_python errors, although I\'ve since stopped assuming that this has anything to do with mod_python. Essentially, I h

Answer 1

"Do these configuration disasters seem like a likely explanation for sporadic ImportErrors"

Yes. An old .pyc file is a disaster of the first magnitude.

We develop on Windows, but run production on Red Hat Linux. An accidentally moved .pyc file is an absolute mystery to debug because (1) it usually runs and (2) it has a Windows filename for the original source, making the traceback error absolutely senseless. I spent hours staring at logs -- on linux -- wondering why the file was "C:\This\N\That".

"If there is still a problem somewhere else in my code, how would I best reproduce it?"

Before reproducing errors, you should try to prevent them.

First, create unit tests to exercise everything.

Start with Django's tests.py testing. Then expand to unittest for all non-Django components. Then write yourself a "run_tests" script that runs every test you own. Run this periodically. Daily isn't often enough.

Second, be sure you're using logging. Heavily.

Third, wrap anything that uses external resources in generic exception-logging blocks like this.

try:
    some_external_resource_processing()
except Exception, e:
    logger.exception( e )
    raise

This will help you pinpoint problems with external resources. Files and databases are often the source of bad behavior due to permission or access problems.

At this point, you have prevented a large number of errors. If you want to run cyclic load testing, that's not a bad idea either. Use unittest for this.

class SomeLoadtest( unittest.TestCase ):
    def test_something( self ):
        self.connection = urllib2.urlopen( "localhost:8000/some/path" )
        results = self.connection.read()

This isn't the best way to do things, but it shows one approach. You might want to start using Selenium to test the web site "from the outside" as a complement to your unittests.