- Trending Categories
Data Structure
Networking
RDBMS
Operating System
Java
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How can I parse a website using Selenium and Beautifulsoup in python?
We can parse a website using Selenium and Beautiful Soup in Python. Web Scraping is a concept used to extract content from the web pages, used extensively in Data Science and metrics preparation. In Python, it is achieved with the BeautifulSoup package.
To have BeautifulSoup along with Selenium, we should run the command −
pip install bs4 selenium
Let us scrap the below links appearing on the page −
Then investigate the html structure of the above elements −
Example
from selenium import webdriver from bs4 import BeautifulSoup #path of chromedriver.exe driver = webdriver.Chrome (executable_path="C:\chromedriver.exe") #launch browser driver.get ("https://www.tutorialspoint.com/about/about_careers.htm") #content whole page in html format s = BeautifulSoup(driver.page_source, 'html.parser') #access to specific ul element with BeautifulSoup methods l = s.find('ul', {'class':'toc chapters'}) #get all li elements under ul rs = l.findAll('li') for r in rs: #get text of li elements print(r.text)
Output
- Related Articles
- How can BeautifulSoup package be used to parse data from a webpage in Python?
- How can BeautifulSoup be used to extract ‘href’ links from a website?
- How can BeautifulSoup package be used to extract the name of the domain of the website in Python?
- How can I delete an element in Selenium using Python?
- How can I scroll a web page using selenium webdriver in python?
- How can I clear text of a textbox using Python Selenium WebDriver?
- How can titles from a webpage be extracted using BeautifulSoup?
- How to get the text from a website using selenium?
- How to remove empty tags using BeautifulSoup in Python?
- Selenium versus BeautifulSoup for Web Scraping.
- How can I perform mouse hover action in Selenium-Python?
- How can I manually set proxy settings in Python Selenium?
- How can I download a file on a click event using selenium?
- How can I close a specific window using Selenium WebDriver with Java?
- How can I select date from a datepicker div using Selenium Webdriver?

Advertisements