What is the best way to run all Python files in a directory?

Running all Python files in a directory simultaneously can save time and effort. Python offers several approaches including subprocess for isolation, exec() for in-process execution, shell scripts for system-level control, and multiprocessing for parallel execution.

Using subprocess.run()

The subprocess module runs Python scripts as separate processes, ensuring isolation between them. This is the safest method as each script runs independently.

Example

Here's how to execute all Python files in a directory using subprocess ?

import subprocess
from pathlib import Path

# Define the directory containing Python files
directory = Path("./scripts")

# Loop through all Python files and execute them
for file in directory.glob("*.py"):
    print(f"Running: {file}")
    subprocess.run(["python", str(file)], check=True)

Running: scripts/example.py
Welcome to Tutorialspoint.
Running: scripts/sample.py
Happy Learning With Tutorialspoint

Using exec() Function

The exec() function executes Python code within the same process. This method is faster but less isolated since all scripts share the same namespace.

Example

Here's how to use exec() to run all Python files ?

from pathlib import Path

directory = Path("./scripts")

# Loop through and execute scripts within the same process
for file in directory.glob("*.py"):
    print(f"Executing: {file}")
    with open(file) as f:
        exec(f.read())

Executing: scripts/example.py
Welcome to Tutorialspoint.
Executing: scripts/sample.py
Happy Learning With Tutorialspoint

Using Shell Script

A shell script provides system-level control to run Python files from outside Python. This approach works well in Unix-based environments like Linux and macOS.

Example

Create a bash script to execute all Python files ?

#!/bin/bash
# Directory containing Python scripts
directory="./scripts"

# Loop through all Python files and execute them
for f in "$directory"/*.py; do
    echo "Running: $f"
    python "$f"
done

Running: ./scripts/example.py
Welcome to Tutorialspoint.
Running: ./scripts/sample.py
Happy Learning With Tutorialspoint

Using Multiprocessing

The multiprocessing module runs multiple Python files in parallel, significantly speeding up execution. Each script runs in its own process for better performance and isolation.

Example

Here's how to use multiprocessing for parallel execution ?

import subprocess
import multiprocessing
from pathlib import Path

# Directory containing Python scripts
directory = Path("./scripts")

# Function to run each script
def run_script(file):
    print(f"Running: {file}")
    subprocess.run(["python", str(file)], check=True)

# Use multiprocessing to run scripts in parallel
if __name__ == "__main__":
    python_files = list(directory.glob("*.py"))
    with multiprocessing.Pool() as pool:
        pool.map(run_script, python_files)

Running: scripts/example.py
Running: scripts/sample.py
Welcome to Tutorialspoint.
Happy Learning With Tutorialspoint

Comparison

Conclusion

Use subprocess.run() for safe isolated execution, multiprocessing for parallel performance, and exec() when scripts need to share data. Choose based on your security, performance, and isolation requirements.