Selenium versus BeautifulSoup for Web Scraping.

We can perform web scraping with Selenium webdriver and BeautifulSoup. Web Scraping is used to extract content from a page. In Python, it is achieved with the BeautifulSoup package.

Let us scrap and get the below links on a page −

Let us also see the html structure of the above links −

Let us see how to do web scraping with BeautifulSoup

To install the required package for Beautifulsoup, we should run the below commands −

pip install bs4
pip install requests

Example

from bs4 import BeautifulSoup
import requests
#get all response
d=requests.get("https://www.tutorialspoint.com/about/about_careers.htm")
#response content whole page in html format
s = BeautifulSoup(d.content, 'html.parser')
#access to specific ul element with BeautifulSoup methods
l = s.find('ul', {'class':'toc reading'})
#access all children of ul
rs = l.findAll('li')
   for r in rs:
      #get text of li elements
         print(r.text)

Now, let us see how to do web scraping with Selenium along with BeautifulSoup.

To have BeautifulSoup along with Selenium, we should run the command −

pip install bs4 selenium

Example

from selenium import webdriver
from bs4 import BeautifulSoup
#path of chromedriver.exe
driver = webdriver.Chrome (executable_path="C:\chromedriver.exe")
#launch browser
driver.get ("https://www.tutorialspoint.com/about/about_careers.htm")
#content whole page in html format
s = BeautifulSoup(driver.page_source, 'html.parser')
#access to specific ul element with BeautifulSoup methods
l = s.find('ul', {'class':'toc reading'})
#get all li elements under ul
rs = l.findAll('li')
for r in rs:
   #get text of li elements
      print(r.text)

Output