Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How to parse local HTML file in Python?
Parsing local HTML files in Python is a common task when dealing with web scraping, data analysis, and automation.
In this article, we will learn how to parse local HTML files in Python. We will explore various techniques to extract data from an HTML file using Python. We will cover modifying and removing elements from the file, printing data, using recursive child generators to traverse the file's structure, finding tag children, and even web scraping by extracting information from a given link. Through code examples and syntax, we will demonstrate how to leverage Python libraries such as BeautifulSoup and lxml to accomplish these tasks efficiently.
Setting up the Environment
Before we dive into parsing HTML files, let's ensure our Python environment has the necessary libraries installed. We will primarily rely on two popular libraries: BeautifulSoup and lxml. To install them, use the following pip commands:
pip install beautifulsoup4 pip install lxml
Once installed, we can begin parsing local HTML files and extracting data. We can do multiple techniques like modifying files, traversing HTML structure, web scraping, etc. Let’s see some of them in detail with syntax and complete examples:
Loading and Modifying HTML files
To parse an HTML file, we need to load it into our Python script. We can achieve this by opening the file using the built−in open function and then reading its contents. Here's an example:
Syntax
with open('example.html', 'r') as file:
html_content = file.read()
Once the HTML file is loaded, we can modify its contents using string manipulation techniques or more advanced methods provided by libraries like BeautifulSoup. For instance, to remove a specific element from the HTML file, we can use BeautifulSoup's extract method:
Input HTML file
#myhtml.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document</title>
</head>
<body>
<div class="my-class">
Hello World
</div>
</body>
</html>
Example
In this example, we loaded the HTML file ('myhtml.html'), created a BeautifulSoup object, found the element to remove using its tag and attributes, and finally removed it from the HTML structure. The modified HTML can be printed using the prettify method to visualize the changes.
from bs4 import BeautifulSoup
# Load the HTML file
with open('myhtml.html', 'r') as file:
html_content = file.read()
# Create a BeautifulSoup object
soup = BeautifulSoup(html_content, 'lxml')
# Find the element to remove by its tag and remove it
element_to_remove = soup.find('div', {'class': 'my-class'})
element_to_remove.extract()
# Print the modified HTML
print(soup.prettify())
Output
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"/> <meta content="IE=edge" http-equiv="X-UA-Compatible"/> <meta content="width=device-width, initial-scale=1.0" name="viewport"/> <title> Document </title> </head> <body> </body> </html>
Extracting Data from HTML Files
Printing or extracting specific data from an HTML file involves navigating its structure. BeautifulSoup provides a range of methods to accomplish this. To extract data, we often need to find the desired element or elements using their tags, classes, or attributes.
For instance, let's consider an HTML file containing a list of articles with the following structure:
Example
In this example, we loaded the HTML file, created a BeautifulSoup object, found the ul element, and then extracted all li elements within it. Finally, we printed the text content of each li element, which represents the article titles.
HTML
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document</title>
</head>
<body>
<div class="">
<ul>
<li>Article 1</li>
<li>Article 2</li>
<li>Article 3</li>
</ul>
</div>
</body>
</html>
Python
from bs4 import BeautifulSoup
# Load the HTML file
with open('myhtml.html', 'r') as file:
html_content = file.read()
# Create a BeautifulSoup object
soup = BeautifulSoup(html_content, 'lxml')
# Find all li elements within the ul tag
articles = soup.find('ul').find_all('li')
# Print the article titles
for article in articles:
print(article.text)
Output
Article 1 Article 2 Article 3
Traversing the HTML Structure with Recursive Child Generators
Recursive child generators are a powerful technique to traverse an HTML file's structure. BeautifulSoup allows us to iterate over the children of a tag using the .children attribute. We can recursively traverse the entire structure to extract the desired information.
Example
In this example, we loaded the HTML file, created a BeautifulSoup object, defined a recursive function traverse_tags, and called it with the root element (in this case, the soup object). The function prints the tag name and its content and then recursively calls itself for each child element.
HTML
myhtml.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document</title>
</head>
<body>
<div class="container">
<h1>Welcome to Tutorialspoint</h1>
<p>Arrays </p>
<p>Linkedin List</p>
</div>
</body>
</html>
Python
from bs4 import BeautifulSoup
# Load the HTML file
with open('myhtml.html', 'r') as file:
html_content = file.read()
# Create a BeautifulSoup object
soup = BeautifulSoup(html_content, 'lxml')
# Define a recursive function to traverse the structure
def traverse_tags(element):
print(element.name)
print(element.text)
for child in element.children:
if child.name:
traverse_tags(child)
# Traverse the HTML structure
traverse_tags(soup)
Output
[document] Document Welcome to Tutorialspoint Arrays Linkedin List html Document Welcome to Tutorialspoint Arrays Linkedin List head Document meta meta meta title Document body Welcome to Tutorialspoint Arrays Linkedin List div Welcome to Tutorialspoint Arrays Linkedin List h1 Welcome to Tutorialspoint p Arrays p Linkedin List
Web Scraping from a Link
In addition to parsing local HTML files, we can also extract useful information by scraping web pages. Using Python libraries such as BeautifulSoup and requests, we can fetch the HTML content of a webpage and extract relevant data.
Syntax
# Define the URL url = 'https://www.tutorialspoint.com/index.htm' # Send a GET request response = requests.get(url) # Create a BeautifulSoup object with the webpage content soup = BeautifulSoup(response.content, 'lxml')
Example
In this example, we used the requests library to send a GET request to the desired webpage. Then, we created a BeautifulSoup object with the response content and extracted the article titles and descriptions using appropriate tags. Finally, we printed the extracted information.
import requests
from bs4 import BeautifulSoup
# Define the URL of the webpage to scrape
url = 'https://www.tutorialspoint.com/index.htm'
# Send a GET request to the webpage
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
print("Fetch was successful.")
# Create a BeautifulSoup object with the webpage content
soup = BeautifulSoup(response.content, 'lxml')
# Find and print the title of the webpage
mytitle = soup.find('title').text
print(f"HTMl Webpage Title: {mytitle}")
# Find and print the first paragraph of the content
myparagraph = soup.find('p').text
print(f"First Paragraph listed in the website: {myparagraph}")
else:
print(f"Error code: {response.status_code}")
Output
Fetch was successful. HTMl Webpage Title: Online Courses and eBooks Library | Tutorialspoint First Paragraph listed in the website: Premium Courses
Conclusion
Parsing local HTML files in Python offers a wide range of possibilities for data extraction and manipulation. We can effectively extract pertinent information from HTML files by modifying the file, removing elements, printing data, utilizing recursive child generators, and web scraping from links. Python makes use of powerful libraries like BeautifulSoup and lxml to navigate and manipulate HTML structures. You can now confidently extract and use data from HTML files in your Python projects with the knowledge and code examples in this article.
1K+ Views
