Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. This file is a simple text-based access control system for computer programs that automatically access web resources. Such programs are called spiders, crawlers, etc. The file specifies the user agent identifier followed by a list of URLs the agent may not access.For example#robots.txt Sitemap: https://example.com/sitemap.xml User-agent: * Disallow: /admin/ Disallow: /downloads/ Disallow: /media/ Disallow: /static/This file is usually put in the top-level directory of your web server.Python's urllib.robotparser module provides RobotFileParser class. It answers questions about whether ... Read More

This module provides a standard interface to break Uniform Resource Locator (URL) strings in components or to combine the components back into a URL string. It also has functions to convert a "relative URL" to an absolute URL given a "base URL."This module supports the following URL schemes -fileftpgopherhdlhttphttpsimapmailtommsnewsnntpprosperorsyncrtsprtspusftpshttpsipsipssnewssvnsvn+sshtelnetwaiswswssurlparse()This function parses a URL into six components, returning a 6-tuple. This corresponds to the general structure of a URL. Each tuple item is a string. The components are not broken up in smaller parts (for example, the network location is a single string), and % escapes are not expanded. The return ... Read More

The HTMLParser class defined in this module provides functionality to parse HTML and XHMTL documents. This class contains handler methods that can identify tags, data, comments and other HTML elements.We have to define a new class that inherits HTMLParser class and submit HTML text using feed() method.from html.parser import HTMLParser class parser(HTMLParser): pass p = parser() p.feed('')We have to override its following methodshandle_starttag(tag, attrs):HTML tags normally are in pairs of starting tag and end tag. For example and . This method is called to handle the start of a tag.Name of the tag converted to lower case. The attrs ... Read More

Function in Python is said to be of higher order. It means that it can be passed as argument to another function and/or can return other function as well. The functools module provides important utilities for such higher order functions.partial() functionThis function returns a callable 'partial' object. The object itself behaves like a function. The partial() function receives another function as argument and freezes some portion of a function’s arguments resulting in a new object with a simplified signature.The built-in int() function converts a number to a decimal integer. Default signature of int() isint(x, base = 10)The partial() function can ... Read More

What is a Copy Operation in Python? Copying in Python refers to the process of creating a duplicate of existing data. The simplest way to create a reference to an object is by using the assignment operator (=), but this does not create an actual copy—it only makes the new variable point to the same object in memory. To create an independent copy, Python provides two methods: shallow copy and deep copy, which can be achieved using the copy module. The Assignment operator doesn't create a new object; instead, it binds a new variable to the same memory address ... Read More

Performing sort operations after every insertion on a long list may be expensive in terms of time consumed by processor. The bisect module ensures that the list remains automatically sorted after insertion. For this purpose, it uses bisection algorithm. The module has following functions:bisect_left()This method locates insertion point for a given element in the list to maintain sorted order. If it is already present in the list, the insertion point will be before (to the left of) any existing entries. The return value caan be used as the first parameter to list.insert()bisect_right()This method is similar to bisect_left(), but returns an ... Read More

Array is a very popular data structure in C/C++, Java etc. In these languages array is defined as a collection of more than one elements of similar data type. Python doesn't have any built-in equivalent of array. It's List as well as Tuple is a collection of elements but they may of different types.Python's array module emulates C type array. The module defines 'array' class. Following constructor creates an array object:array(typecode, initializer)The typecode argument determines the type of array. Initializer should be a sequence with all elements of matching type.Following statement creates an integer array object:>>> import array >>> arr ... Read More

Python developers can extend and modify the behavior of a callable functions, methods or classes without permanently modifying the callable itself by using decorators. In short we can say they are callable objects which are used to modify functions or classes.Function decorators are functions which accepts function references as arguments and adds a wrapper around them and returns the function with the wrapper as a new function.Let’s understand function decorator bye an example:Code1@decorator def func(arg):    return "value"Above code is same as:Code2def func(arg):    return "value" func = decorator(func)So from above, we can see a decorator is simply another function ... Read More

Python provides two built-ins functions for “AND” and “OR” operations are All and Any functions.Python any() functionThe any() function returns True if any item in an iterable are true, otherwise it returns False. However, if the iterable object is empty, the any () function will return False.Syntaxany(iterable)The iterable object can be a list, tuple or dictionary.Example 1>>> mylst = [ False, True, False] >>> x = any(mylst) >>> x TrueOutputOutput is True because the second item is True.Example 2Tuple – check if any item is True>>> #Tuple - check if any item is True >>> mytuple = (0, 1, 0, ... Read More

Though we all know python is not as fast or efficient as other complied languages. However, there are many big companies which shows us that python code can handle much bigger workload which shows it’s not that slow. In this section, we are going to see some of the tips that one should keep in mind so that a correct python program runs even faster and more efficient.Tip 1: Go for built-in functionsThough we can write efficient code in python, but it’s very hard to beat built-in functions(which are written in C). Below image shows the list of python built-in ... Read More