How can BeautifulSoup be used to extract ‘href’ links from a website?

BeautifulSoup is a third party Python library that is used to parse data from web pages. It helps in web scraping, which is a process of extracting, using, and manipulating the data from different resources.

Web scraping can also be used to extract data for research purposes, understand/compare market trends, perform SEO monitoring, and so on.

The below line can be run to install BeautifulSoup on Windows −

pip install beautifulsoup4

Following is an example −

Example

from bs4 import BeautifulSoup
import requests
url = "https://en.wikipedia.org/wiki/Algorithm"
req = requests.get(url)
soup = BeautifulSoup(req.text, "html.parser")
print("The href links are :")
for link in soup.find_all('a'):
   print(link.get('href'))

Output

The href links are :
…
https://stats.wikimedia.org/#/en.wikipedia.org
https://foundation.wikimedia.org/wiki/Cookie_statement
https://wikimediafoundation.org/
https://www.mediawiki.org/

Explanation

• The required packages are imported, and aliased.

• The website is defined.

• The url is opened, and data is read from it.

• The ‘BeautifulSoup’ function is used to extract text from the webpage.

• The ‘find_all’ function is used to extract text from the webpage data.

• The href links are printed on the console.

AmitDiwan

Published on 18-Jan-2021 12:53:53

• Related Questions & Answers
• How can BeautifulSoup package be used to extract the name of the domain of the website in Python?
• How to extract website name from their links in R?
• How can BeautifulSoup package be used to parse data from a webpage in Python?
• How can ‘placeholders’ in Tensorflow be used while multiplying matrices?
• How can titles from a webpage be extracted using BeautifulSoup?
• How can the ‘subplot’ function be used to create two graphs in Matplotlib Python?
• Python program to extract ‘k’ bits from a given position?
• Java program to extract ‘k’ bits from a given position
• How can I parse a website using Selenium and Beautifulsoup in python?
• What are all the ways keyword ‘this’ can be used in Java?
• How can the ‘Word2Vec’ algorithm be trained using Tensorflow?
• How can Keras be used to extract features from only one layer of the model using Python?
• Validate input: replace all ‘a’ with ‘@’ and ‘i’ with ‘!’JavaScript
• How can data be represented visually using ‘seaborn’ library in Python?
• Explain how the top ‘n’ elements can be accessed from series data structure in Python?