How to scan through a directory recursively in Python?

A directory is simply defined as a collection of subdirectories and files. A directory hierarchy is constructed by organizing all the files and subdirectories within a main directory, also known as the root directory. These subdirectories are separated using a / operator in a directory hierarchy.

In Python, we have different methods to scan through a directory recursively. In this article we are going to see about each method:

• Using os.walk() method
• Using glob.glob() method
• Using os.listdir() method with recursion

Note: The directories are handled by the operating system; therefore, whenever one needs a status update on any directory it needs to be done using the os module.

Using os.walk() Method

The os.walk() function generates file names in a directory tree by walking it top-down or bottom-up. It returns a three-tuple for each directory in the tree rooted at directory top: (root, dirs, files).

Example 1 - Basic Directory Walking

Here, we are using the os.walk() method to display all the files and subdirectories present in the current root directory ?

import os

path = "."
for root, d_names, f_names in os.walk(path):
    print(f"Directory: {root}")
    print(f"Subdirectories: {d_names}")
    print(f"Files: {f_names}")
    print("---")

Directory: .
Subdirectories: ['docs', 'src']
Files: ['main.py', 'README.md']
---
Directory: ./docs
Subdirectories: []
Files: ['tutorial.txt']
---
Directory: ./src
Subdirectories: []
Files: ['helper.py', 'config.py']
---

Example 2 - Getting Full File Paths

We can also make use of the full path for each file to scan through a directory. For that, we use the os.path.join() method to create complete file paths ?

import os

path = "."
file_paths = []

for root, d_names, f_names in os.walk(path):
    for f in f_names:
        file_paths.append(os.path.join(root, f))

print("All file paths:")
for file_path in file_paths:
    print(file_path)

All file paths:
./main.py
./README.md
./docs/tutorial.txt
./src/helper.py
./src/config.py

Using glob.glob() Method

The glob module is used to get the pathnames matching a specific pattern. The glob.glob() method searches for all pathnames containing the given path specification. Using ** enables recursive matching.

Example - Recursive Pattern Matching

In this example, we use glob() with the ** pattern to recursively find all Python files ?

import glob

# Find all Python files recursively
python_files = glob.glob("**/*.py", recursive=True)
print("Python files found:")
for file in python_files:
    print(file)

# Find all files recursively
all_files = glob.glob("**/*", recursive=True)
print("\nAll files and directories:")
for item in all_files[:10]:  # Show first 10 items
    print(item)

Python files found:
main.py
src/helper.py
src/config.py

All files and directories:
README.md
main.py
docs
src
docs/tutorial.txt
src/helper.py
src/config.py

Using os.listdir() with Recursion

The os.listdir() method lists all files and directories in a specified directory. To make it recursive, we need to implement our own recursive function.

Example - Custom Recursive Function

In this example, we create a recursive function using os.listdir() to scan through directories ?

import os

def scan_directory_recursive(path):
    items = []
    try:
        for item in os.listdir(path):
            item_path = os.path.join(path, item)
            items.append(item_path)
            
            if os.path.isdir(item_path):
                items.extend(scan_directory_recursive(item_path))
    except PermissionError:
        print(f"Permission denied: {path}")
    
    return items

# Scan current directory recursively
all_items = scan_directory_recursive(".")
print("All items found:")
for item in all_items:
    print(item)

All items found:
./README.md
./main.py
./docs
./docs/tutorial.txt
./src
./src/helper.py
./src/config.py

Comparison

Conclusion

Use os.walk() for simple recursive directory traversal, glob.glob() for pattern-based file searching, and custom recursive functions with os.listdir() when you need specialized directory processing logic.