Heroku app crashed after pushing small change. Very confused

Question

I am using Heroku with python and Flask. My app was working fine until I updated a few lines in my python application file. The app runs fine locally, but I now have the fol

Answer 1

I was able to fix the problem, but still dont know why it occurred in the first place!

After a lot of experimentation, I ended up setting up a completely new app on Heroku. I checked the environment variables in the new app and got the following:

 Cinnas-MacBook-Pro:thawing-temple-4323 cinna$ heroku config
 === thawing-temple-4323 Config Vars
 FACEBOOK_APP_ID:  ***
 FACEBOOK_SECRET:  ***
 PATH:             bin:/usr/local/bin:/usr/bin:/bin
 PYTHONUNBUFFERED: true

Checking my original app (the broken one), I realized that new environment variables were somehow added in my last push as indicated by my logs:

 2012-10-04T04:20:04+00:00 heroku[api]: Add PYTHONUNBUFFERED, PYTHONPATH, PYTHONHOME, LANG, LD_LIBRARY_PATH, PATH, PYTHONHASHSEED, LIBRARY_PATH config by ***@***

and by checking my environment variables:

 Cinnas-MacBook-Pro:infinite-fortress-4866 cinna$ heroku config
 === infinite-fortress-4866 Config Vars
 LANG:             en_US.UTF-8 
 LD_LIBRARY_PATH:  /app/.heroku/vendor/lib
 LIBRARY_PATH:     /app/.heroku/vendor/lib
 PATH:             /app/.heroku/venv/bin:/bin:/usr/local/bin:/usr/bin   
 PYTHONHASHSEED:   random
 PYTHONHOME:       /app/.heroku/venv/
 PYTHONPATH:       /app/
 PYTHONUNBUFFERED: true

I removed these new variables with the command:

 heroku config:remove PYTHONPATH PYTHONHOME LANG LD_LIBRARY_PATH PYTHONHASHSEED LIBRARY_PATH

and my app started to work again. I've been pushing more code, and this problem has not occurred again.

I am still really curious why/how these variables were added in the first place since all I did was do a git push.

Answer 2

This bit me, too. I'm not sure when it started. I don't believe it was in response to any change on my part, I just noticed Application Error on my site today, and found this in the logs:

2012-12-12T16:02:06+00:00 heroku[web.1]: State changed from crashed to starting
2012-12-12T16:02:09+00:00 heroku[web.1]: Starting process with command `aspen --network_address=:40856 --www_root=doc/ --project_root=doc/.aspen`
2012-12-12T16:02:10+00:00 app[web.1]: ImportError: No module named site
2012-12-12T16:02:11+00:00 heroku[web.1]: Process exited with status 1
2012-12-12T16:02:11+00:00 heroku[web.1]: State changed from starting to crashed

I had another release ready to go, so I just deployed as usual. After a git push heroku, the site is back up. My heroku config doesn't have the extra envvars listed above:

$ heroku config
=== aspen-io Config Vars
ASPEN_IO_SHOW_GA: yes
PATH:             bin:/usr/local/bin:/usr/bin:/bin
PYTHONUNBUFFERED: true

Update: @kennethreitz pointed me to this:

Cannot import module site

This occurs when the configured environment variables don't match the paths of the installed Python. When this occurs, it is because someone purged an app's cache without understanding the above implications.

To fix, either purge the cache and update the configuration, or restore the expected configurations (preferred).

Answer 3

I experienced a very similar problem yesterday (6th Dec. 2012). Out of the blue, every python invocation died with 'ImportError: No module named site'. Heroku support got back to me today and they say it's fixed on their end, so the following workaround shouldn't be required. I'll leave this here in case it helps someone else diagnose.

I checked my heroku config vars though, and there were no PYTHON* variables set. They were set as env vars at the shell level though:

$ heroku run set | grep PYTHON
PYTHONHASHSEED=random
PYTHONHOME=/app/.heroku/venv/
PYTHONPATH=/app/
PYTHONUNBUFFERED=true 

/app/.heroku/venv was a non-existent directory. If I overrode PYTHONHOME with a config var, and pointed to where my virtualenv actually was, it all started working again:

$ heroku config:set PYTHONHOME=/app

/app appears to be a mount point for the project root directory. Digging through the history of the Python buildpack, it looks like when I started my project, everyone made their virtualenvs in the project root. Now new projects make virtualenvs in a venv/ subdirectory. Support said they were gradually rolling out a buildpack change, and I guess the checks for the old way of doing things didn't kick in for me.

Here's where to look for the buildpack internals: https://github.com/heroku/heroku-buildpack-python/blob/master/bin/compile