Iterate over lines from multiple input streams in Python

Python’s built-in open() function opens one file in read/write mode and read/write operations on it. To perform processing on multiple files in a batch, one has to use fileinput module of Python’s standard library. This module provides a Fileinput class with functionality of iterating over files. The module also defines helper functions for the same purpose.

Primary interface to this module is input() function. This function returns instance of Fileinput class.

fileinput.input(files, inplace, mode)

The files parameter is name of one or more files to be read one by one. Each file acts as a generator and using a for loop it can be iterated upon. Each line in the file will be printed on Python console.

>>> for line in fileinput.input('data.txt'):
print (line)

The files parameter can be a tuple consisting of many files. Contents of files will be displayed one by one.

>>> for line in fileinput.input(files=('a.txt', 'b.txt')):
print (line)

The Fileinput class can also be used a context manager in the with statement.

>>> with fileinput.input(files=('a.txt', 'b.txt')) as f:
for line in f:
print (line)

The fileinput module has following functions defined in it.

Following statement prints each line in the file along with the line number

>>> for line in fileinput.input('books.py'):
print ('{}->{}'.format(fileinput.filelineno(), line))

The sample output of above code is

1->import sqlite3
2->conn = sqlite3.connect('c:/python36/books.db')
3->cursor = conn.cursor()
4->cursor.execute("SELECT * from books;")
5->print(cursor.fetchall())

Following code print each file name in folder followed by numbered lines in it. In this program glob() function is used which returns list of files in current path optionally with matching wild cards. Here glob(‘*.py’) will return list of all files with .py extension in the current folder. This list is used as files parameter to fileinput.input() function.

import fileinput, glob, sys
for line in fileinput.input(glob.glob("*.py")):
if fileinput.isfirstline():
print (fileinput.filename(),'>')
sys.stdout.write ("{}.{}".format(fileinput.filelineno(),line))

Note the use of isfirstline() function. When iteration of new file starts, this function returns true and file name as returned by fileinput.filename() function is printed first and then the lines with numbers are displayed. For example

1.py >
1.a = 10
2.b = 20
3.print ('addition=',a+b)
hello.py >
1.x = 10
2.y = 20
3.z = x+y
4.print ("x+y=",z)

inplace parameter

By default, inplace = False for fileinput.input() function. If it is set to True, it makes the input file writable.

Assuming that there is a ‘msg.txt’ with following text in it.

Hello Python. Good morning

Following code opens the file using fileinput module and modifies its contents in place.

>>> for line in fileinput.input(files='msg.txt',inplace = True):
line = line.replace('morning', 'evening')
sys.stdout.write(line)

The ‘msg.txt’ will show the changes done.