- Python Web Scraping Tutorial
- Python Web Scraping - Home
- Introduction
- Getting Started with Python
- Python Modules for Web Scraping
- Legality of Web Scraping
- Data Extraction
- Data Processing
- Processing Images and Videos
- Dealing with Text
- Scraping Dynamic Websites
- Scraping Form based Websites
- Processing CAPTCHA
- Testing with Scrapers
- Python Web Scraping Resources
- Python Web Scraping - Quick Guide
- Python Web Scraping - Resources
- Python Web Scraping - Discussion
Getting Started with Python
In the first chapter, we have learnt what web scraping is all about. In this chapter, let us see how to implement web scraping using Python.
Why Python for Web Scraping?
Python is a popular tool for implementing web scraping. Python programming language is also used for other useful projects related to cyber security, penetration testing as well as digital forensic applications. Using the base programming of Python, web scraping can be performed without using any other third party tool.
Python programming language is gaining huge popularity and the reasons that make Python a good fit for web scraping projects are as below −
Syntax Simplicity
Python has the simplest structure when compared to other programming languages. This feature of Python makes the testing easier and a developer can focus more on programming.
Inbuilt Modules
Another reason for using Python for web scraping is the inbuilt as well as external useful libraries it possesses. We can perform many implementations related to web scraping by using Python as the base for programming.
Open Source Programming Language
Python has huge support from the community because it is an open source programming language.
Wide range of Applications
Python can be used for various programming tasks ranging from small shell scripts to enterprise web applications.
Installation of Python
Python distribution is available for platforms like Windows, MAC and Unix/Linux. We need to download only the binary code applicable for our platform to install Python. But in case if the binary code for our platform is not available, we must have a C compiler so that source code can be compiled manually.
We can install Python on various platforms as follows −
Installing Python on Unix and Linux
You need to followings steps given below to install Python on Unix/Linux machines −
Step 1 − Go to the link https://www.python.org/downloads/
Step 2 − Download the zipped source code available for Unix/Linux on above link.
Step 3 − Extract the files onto your computer.
Step 4 − Use the following commands to complete the installation −
run ./configure script make make install
You can find installed Python at the standard location /usr/local/bin and its libraries at /usr/local/lib/pythonXX, where XX is the version of Python.
Installing Python on Windows
You need to followings steps given below to install Python on Windows machines −
Step 1 − Go to the link https://www.python.org/downloads/
Step 2 − Download the Windows installer python-XYZ.msi file, where XYZ is the version we need to install.
Step 3 − Now, save the installer file to your local machine and run the MSI file.
Step 4 − At last, run the downloaded file to bring up the Python install wizard.
Installing Python on Macintosh
We must use Homebrew for installing Python 3 on Mac OS X. Homebrew is easy to install and a great package installer.
Homebrew can also be installed by using the following command −
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
For updating the package manager, we can use the following command −
$ brew update
With the help of the following command, we can install Python3 on our MAC machine −
$ brew install python3
Setting Up the PATH
You can use the following instructions to set up the path on various environments −
Setting Up the Path on Unix/Linux
Use the following commands for setting up paths using various command shells −
For csh shell
setenv PATH "$PATH:/usr/local/bin/python".
For bash shell (Linux)
ATH="$PATH:/usr/local/bin/python".
For sh or ksh shell
PATH="$PATH:/usr/local/bin/python".
Setting Up the Path on Windows
For setting the path on Windows, we can use the path %path%;C:\Python at the command prompt and then press Enter.
Running Python
We can start Python using any of the following three ways −
Interactive Interpreter
An operating system such as UNIX and DOS that is providing a command-line interpreter or shell can be used for starting Python.
We can start coding in interactive interpreter as follows −
Step 1 − Enter python at the command line.
Step 2 − Then, we can start coding right away in the interactive interpreter.
$python # Unix/Linux or python% # Unix/Linux or C:> python # Windows/DOS
Script from the Command-line
We can execute a Python script at command line by invoking the interpreter. It can be understood as follows −
$python script.py # Unix/Linux or python% script.py # Unix/Linux or C: >python script.py # Windows/DOS
Integrated Development Environment
We can also run Python from GUI environment if the system is having GUI application that is supporting Python. Some IDEs that support Python on various platforms are given below −
IDE for UNIX − UNIX, for Python, has IDLE IDE.
IDE for Windows − Windows has PythonWin IDE which has GUI too.
IDE for Macintosh − Macintosh has IDLE IDE which is downloadable as either MacBinary or BinHex'd files from the main website.