Scrapy - Environment

In this chapter, we will discuss how to install and set up Scrapy. Scrapy must be installed with Python.

Scrapy can be installed by using pip. To install, run the following command −

pip install Scrapy

Windows

Note − Python 3 is not supported on Windows OS.

Step 1 − Install Python 2.7 from Python

Set environmental variables by adding the following paths to the PATH −

C:\Python27\;C:\Python27\Scripts\;

You can check the Python version using the following command −

python --version

Step 2 − Install OpenSSL.

Add C:\OpenSSL-Win32\bin in your environmental variables.

Note − OpenSSL comes preinstalled in all operating systems except Windows.

Step 3 − Install Visual C++ 2008 redistributables.

Step 4 − Install pywin32.

Step 5 − Install pip for Python versions older than 2.7.9.

You can check the pip version using the following command −

pip --version

Step 6 − To install scrapy, run the following command −

pip install Scrapy

If you have anaconda or miniconda installed on your machine, run the below command to install Scrapy using conda −

conda install -c scrapinghub scrapy

Scrapinghub company supports official conda packages for Linux, Windows, and OS X.

Note − It is recommended to install Scrapy using the above command if you have issues installing via pip.

The latest version of Python is pre-installed on Ubuntu OS. Use the Ubuntu packages aptgettable provided by Scrapinghub. To use the packages −

Step 1 − You need to import the GPG key used to sign Scrapy packages into APT keyring −

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7

Step 2 − Next, use the following command to create /etc/apt/sources.list.d/scrapy.list file −

echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee 
/etc/apt/sources.list.d/scrapy.list

Step 3 − Update package list and install scrapy −

sudo apt-get update && sudo apt-get install scrapy

You can install Scrapy from AUR Scrapy package using the following command −

yaourt -S scrapy

Use the following command to install Xcode command line tools −

xcode-select --install

Instead of using system Python, install a new updated version that doesn't conflict with the rest of your system.

Step 1 − Install homebrew.

Step 2 − Set environmental PATH variable to specify that homebrew packages should be used before system packages −

echo "export PATH = /usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc

Step 3 − To make sure the changes are done, reload .bashrc using the following command −

source ~/.bashrc

Step 4 − Next, install Python using the following command −

brew install python

Step 5 − Install Scrapy using the following command −

pip install Scrapy