TwitterFacebookPinterestGoogle+

Installing Guide of Scrapy: MAC OS X

Mac OS X

Building Scrapy’s dependencies requires the presence of a C compiler and development headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run:

xcode-select --install

There’s a known issue that prevents pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:

  • (Recommended) Don’t use system python, install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:

    • Install homebrew following the instructions in http://brew.sh/

    • Update your PATH variable to state that homebrew packages should be used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell):

      echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
      
    • Reload .bashrc to ensure the changes have taken place:

      source ~/.bashrc
      
    • Install python:

      brew install python
      
    • Latest versions of python have pip bundled with them so you won’t need to install it separately. If this is not the case, upgrade python:

      brew update; brew upgrade python
      
  • (Optional) Install Scrapy inside an isolated python environment.

    This method is a workaround for the above OS X issue, but it’s an overall good practice for managing dependencies and can complement the first method.

    virtualenv is a tool you can use to create virtual environments in python. We recommended reading a tutorial like http://docs.python-guide.org/en/latest/dev/virtualenvs/ to get started.

After any of these workarounds you should be able to install Scrapy:

pip install Scrapy

Leave a Reply

Your email address will not be published. Required fields are marked *

Sections

Shows

Local News

Tools

About Us

Follow Us

Skip to toolbar