How can I iterate over files in a given directory in Python?

Iterating over files in a given directory helps to perform tasks such as finding files that match certain criteria or counting the number of files in a directory. Python provides several methods to walk through all the existing files in a directory ?

• os.listdir() method
• os.walk() method
• os.scandir() method
• Using pathlib module
• glob.iglob() method

In this article, we will explore all these methods for iterating over files in a given directory in Python.

Using os.listdir() Method

The os.listdir() method returns a list of all files and directories in the specified path. It's simple but doesn't differentiate between files and directories.

Syntax

os.listdir(path)

Example

The following example lists all items in a directory ?

import os

# List all items in current directory
items = os.listdir('.')

print("Items in current directory:")
for item in items:
    print(item)

Items in current directory:
file1.txt
folder1
script.py

Filtering Only Files

To filter only files (not directories), combine with os.path.isfile() ?

import os

directory = '.'
files = [f for f in os.listdir(directory) if os.path.isfile(os.path.join(directory, f))]

print("Files only:")
for file in files:
    print(file)

Files only:
file1.txt
script.py

Using os.walk() Method

The os.walk() method recursively walks through a directory tree, yielding a three-tuple for each directory: (dirpath, dirnames, filenames).

Example

This example walks through a directory tree recursively ?

import os

# Walk through directory tree
for root, dirs, files in os.walk('.'):
    print(f"Directory: {root}")
    print(f"Subdirectories: {dirs}")
    print(f"Files: {files}")
    print("-" * 30)

Directory: .
Subdirectories: ['folder1']
Files: ['file1.txt', 'script.py']
------------------------------
Directory: ./folder1
Subdirectories: []
Files: ['nested_file.txt']
------------------------------

Iterating Only Files

To iterate through all files recursively ?

import os

print("All files recursively:")
for root, dirs, files in os.walk('.'):
    for file in files:
        file_path = os.path.join(root, file)
        print(file_path)

All files recursively:
./file1.txt
./script.py
./folder1/nested_file.txt

Using os.scandir() Method

The os.scandir() method is more efficient than listdir() as it returns DirEntry objects with additional information.

Example

This example uses scandir() to get detailed file information ?

import os

print("Using os.scandir():")
with os.scandir('.') as entries:
    for entry in entries:
        if entry.is_file():
            print(f"File: {entry.name}")
        elif entry.is_dir():
            print(f"Directory: {entry.name}")

Using os.scandir():
File: file1.txt
Directory: folder1
File: script.py

Using pathlib Module

The pathlib module provides an object-oriented approach to handling filesystem paths. It's more modern and readable than os.path.

Example

Using pathlib to iterate through files ?

from pathlib import Path

# Current directory as Path object
directory = Path('.')

print("Files using pathlib:")
for item in directory.iterdir():
    if item.is_file():
        print(f"File: {item.name}")
    elif item.is_dir():
        print(f"Directory: {item.name}")

Files using pathlib:
File: file1.txt
Directory: folder1
File: script.py

Using glob() with pathlib

The glob() method supports pattern matching ?

from pathlib import Path

directory = Path('.')

print("Python files using glob:")
for py_file in directory.glob("*.py"):
    print(py_file.name)

print("\nAll files recursively:")
for file in directory.rglob("*"):
    if file.is_file():
        print(file)

Python files using glob:
script.py

All files recursively:
file1.txt
script.py
folder1/nested_file.txt

Using glob.iglob() Method

The glob.iglob() method returns an iterator that yields paths matching a specified pattern. It's memory-efficient for large directories.

Example

Using glob.iglob() with different patterns ?

import glob

print("All files in current directory:")
for file in glob.iglob("*"):
    print(file)

print("\nOnly .txt files:")
for txt_file in glob.iglob("*.txt"):
    print(txt_file)

print("\nAll files recursively:")
for file in glob.iglob("**/*", recursive=True):
    print(file)

All files in current directory:
file1.txt
folder1
script.py

Only .txt files:
file1.txt

All files recursively:
file1.txt
folder1
script.py
folder1/nested_file.txt

Comparison

Conclusion

Use pathlib for modern Python code with readable syntax. For performance-critical applications, use os.scandir(). Use glob.iglob() when you need pattern matching or memory-efficient iteration over large directories.