In Which A Great Secret Is Revealed

Prepare to be enlightened:

Here’s my tale of woe. I decided to upgrade my Enthought because it’s nice to be able to build things for a 64-bit architecture. However, Enthought decided to change their system to something called “Canopy” which I can only assume is some sort of dumb marketing gimmick. The name is meaningless, but what *does* mean something is that it changed the install path.

Ok, fine, it did do that. However, it relied, for reasons unknown to me, on an old set of build flags which were apparently a carry-over from the previous version of Enthought. When I tried to pip install scikit-learn, I got a bunch of compilation errors related to a missing header. Turns out that I never got these errors before because the missing headers only apply to 64-bit builds, and due to an issue with the Qt libraries not working right in previous versions of Enthought, I had been using the 32-bit version (which itself was problematic because it sometimes caused various version shifts relative to other Python-related tools, but NEVER MIND THAT FOR NOW).

Total bummer, right? What I discovered from reading the logs was that when I ran setup.py install, the CFLAGS parameter being used to build the C components of scikit-learn pointed to /Developer/SDKs/MacOSX10.6.sdk. This is a problem because when you upgrade XCode (which I absolutely had to do to get a different build working), XCode *ALSO* changes the paths to the SDKs! For reasons which are, like the above path change, shrouded in mystery, the SDK for OS X 10.7 and up now lives in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk.

At this dramatic juncture in the narrative, I realized that I needed to tell distutils, which is Python’s build system, where to find the new SDK. But how do I do that? Where does it get its information? Well, to find out *that* carefully concealed secret, you have to go to the distutils directory and look at a file called sysconfig.py. Specifically, take a look at a function called _init_posix(). Therein you will discover that the dictionary from which distutils gets its flags is seeded by reading the top-level Python makefile, which is located in the lib/python2.7/config/ subdirectory of the location where Enthought (excuse me, “Canopy”) installed itself.

Armed with this powerful information, you can now symlink /Developer/SDKs/MacOSX10.7.sdk to the location where the true SDK for 10.7 lives, and edit the Makefile’s CFLAGS options to change the isysroot parameter to the SDK symlink.

And now you can sudo pip scikit-learn install, and it will build without errors. You’re welcome.