Why isn't pip updating my numpy and scipy?

Question

My problem is that pip won\'t update my Python Packages, even though there are no errors.

It is similar to this one, but I am still now sure what to do. Basically, ALL

Answer 1

Old question, but I found it when trying to solve this issue, will post my solution.

I found @abarnert's diagnosis to be correct and helpful, but I don't like any of the solutions: I really want to upgrade the default version of numpy. The challenge is that the directory these guys are in (which @abarnert mentioned) cannot be touched even by sudo, as they are in this "wheel" group. In fact, if you go there and do sudo rm -rf blah, it will give you a permission denied error.

To get around this, we have to take drastic action:

1. Reboot the computer in recovery mode
2. Find the terminal and type csrutil disable
3. Reboot normally, then upgrade numpy with pip2 install --user --upgrade numpy (and same for any other packages that have this problem)
4. Repeat steps a and b, this time changing "disable" to "enable"

Note: "csrutil disable" is serious business that can destabilize your machine, I would use it only when absolutely necessary and re-enable it ASAP. But AFAIK it's the only way to upgrade Python packages in a wheel directory.

Answer 2

Rename the numpy and scipy versions installed by Apple in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/ so it starts using the newer versions installed by Pip.

Answer 3

In OS X 10.9, Apple's Python comes with a bunch of pre-installed extra packages, in a directory named /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python. Including numpy.

And the way they're installed (as if by using easy_install with an ancient pre-0.7 version of setuptools, but not into either of the normal easy_install destinations), pip doesn't know anything about them.

So, what happens is that sudo pip install numpy installs a separate copy of numpy into '/Library/Python/2.7/site-packages'—but in your sys.path, the Extras directory comes before the site-packages directory, so import numpy still finds Apple's copy. I'm not sure why that is, but it's probably not something you want to monkey with.

---

So, how do you fix this?

The two best solutions are:

• Use virtualenv, and install your numpy and friends into a virtual environment, instead of system-wide. This has the downside that you have to learn how to use virtualenv—but that's definitely worth doing at some point, and if you have the time to learn it now, go for it.

• Upgrade to Python 3.x, either from a python.org installer or via Homebrew. Python 3.4 or later comes with pip, and doesn't come with any pip-unfriendly pre-installed packages. And, unlike installing a separate 2.7, it doesn't interfere with Apple's Python at all; python3 and python, pip3 and pip, etc., will all be separate programs, and you don't have to learn anything about how PATH works or any of that. This has the downside that you have to learn Python 3.x, which has some major changes, so again, a bit of a learning curve, but again, definitely worth doing at some point.

---

Assuming neither of those is possible, I think the simplest option is to use easy_install instead of pip, for the packages you want to install newer versions of any of Apple's "extras". You can get a full list of those by looking at what's in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python. When you upgrade numpy, you probably also want to upgrade scipy and matplotlib; I think everything else there is unrelated. (You can of course upgrade PyObjC or dateutil or anything else you care about there, but you don't have to.)

This isn't an ideal solution; there are a lot of reasons easy_install is inferior to pip (e.g., not having an uninstaller, so you're going to have to remember where that /Library/blah/blah path is (or find it again by printout out sys.path from inside Python). I wouldn't normally suggest easy_install for anything except readline and pip itself (and then only with Apple's Python). But in this case, I think it's simpler than the other alternatives.