URL handling Python modules (urllib)

PythonServer Side ProgrammingProgramming

Python language is used extensively for web programming. When we browser website we use the web address which is also known as URL or uniform resource locator. Python has inbuilt materials which can handle the calls to the URL as well as pass the result that comes out of visiting the URL. In this article we will see a module named as urllib. We will also see the various functions present in this module which help in getting the result from the URL.

Installing urllib

To install urllib in the python environment, we use the below command using pip.

pip install urllib

Running the above code gives us the following result −

Opening an URL

The request.urlopen method is used to visit an URL and fetch its content to the python environment.

Example

import urllib.request
address = urllib.request.urlopen('https://www.tutorialspoint.com/')
print(address.read())

Output

Running the above code gives us the following result −

b'<!DOCTYPE html>\r\n<!--[if IE 8]><html class="ie ie8"> <![endif]-->\r\n<!--[if IE 9]><html class……..
……………
……………….
new Date());\r\ngtag(\'config\', \'UA-232293-6\');\r\n</script>\r\n</body>\r\n</html>\r\n' -->

urllib.parse

We can parse the URL to check if it is a valid one or not. We can also Pass a query string to the search option. The response can be checked for its validity and we can print the entire response if it is a valid one.

Example

import urllib.request
import urllib.parse
url='https://tutorialspoint.com'
values= {'q':'python'}
data = urllib.parse.urlencode(values)
data = data.encode('utf-8') # data should be bytes
print(data)
req = urllib.request.Request(url, data)
resp = urllib.request.urlopen(req)
print(resp)
respData = resp.read()
print(respData)

Output

Running the above code gives us the following result −

b'q=python'
<http.client.HTTPResponse object at 0x00000195BF706850>
b'<!DOCTYPE html>\r\n<!--[if IE 8]><html class="ie ie8"> <![endif]…………
…………………
\r\n</script>\r\n</body>\r\n</html<\r\n' -->

urllib.parse.urlsplit

urlsplit can be used to takein an url, then split it into parts which can be used for further data manipulation. For example if we want to programmatically judge if a URL is SSL certified or not then we apply urlsplit and get the scheme value to decide. In the below example we check the different parts of the supplied URL.X

Output

import urllib.parse
url='https://tutorialspoint.com/python'
value = urllib.parse.urlsplit(url)
print(value)

Running the above code gives us the following result −

SplitResult(scheme='https', netloc='tutorialspoint.com', path='/python', query='', fragment='')
raja
Published on 14-Feb-2020 07:39:27
Advertisements