How do I import all the submodules of a Python namespace package?

Namespace packages are a type of package in Python that allows you to split sub-packages and modules within a single package across multiple, separate distribution packages. Unlike normal packages, namespace packages don't require an __init__.py file.

Automatically importing all submodules within a namespace package serves several purposes, like auto-registration without manually importing, and loading all available plugins in a system.

Using pkgutil.iter_modules()

The pkgutil.iter_modules() function finds and lists submodules and subpackages within a given package. It returns an iterator containing ModuleInfo objects, each with information about a found module or package ?

import pkgutil
import importlib

def import_submodules(package_name):
    """Import all submodules of a package, recursively."""
    if isinstance(package_name, str):
        package = importlib.import_module(package_name)
    else:
        package = package_name
        package_name = package.__name__
    
    results = {}
    
    # Check if package has __path__ (required for pkgutil.iter_modules)
    if not hasattr(package, '__path__'):
        return results
    
    for _, name, is_pkg in pkgutil.iter_modules(package.__path__, package_name + '.'):
        try:
            results[name] = importlib.import_module(name)
            
            # Recursively import subpackages
            if is_pkg:
                results.update(import_submodules(name))
        except ImportError as e:
            print(f"Failed to import {name}: {e}")
            
    return results

# Example usage with a standard package
modules = import_submodules('json')
print("Imported modules:", list(modules.keys()))

Imported modules: ['json.encoder', 'json.decoder', 'json.scanner']

This function imports all submodules and subpackages of a given package, returning a dictionary where keys are module names and values are the imported module objects.

Using importlib.resources

The importlib.resources module provides access to resources within a package. This is the modern approach for Python 3.9+ and works effectively with namespace packages ?

import importlib.resources
import importlib.util
import importlib
import types

def import_submodules_with_resources(package_name):
    """
    Import all submodules using importlib.resources (Python 3.9+)
    """
    results = {}
    
    try:
        # Import the package first
        package = importlib.import_module(package_name)
        
        # Skip if not a package
        if not hasattr(package, '__path__'):
            return results
            
        # Get package files
        package_path = importlib.resources.files(package_name)
        
        # Process all items in the package
        for item in package_path.iterdir():
            # Handle Python modules (.py files)
            if item.is_file() and item.name.endswith('.py') and not item.name.startswith('__'):
                module_name = f"{package_name}.{item.name[:-3]}"
                try:
                    module = importlib.import_module(module_name)
                    results[module_name] = module
                except ImportError as e:
                    print(f"Failed to import {module_name}: {e}")
                    
            # Handle subpackages (directories)
            elif item.is_dir() and not item.name.startswith('__'):
                subpackage_name = f"{package_name}.{item.name}"
                
                # Check if it's a valid package
                spec = importlib.util.find_spec(subpackage_name)
                if spec is not None:
                    try:
                        subpackage = importlib.import_module(subpackage_name)
                        results[subpackage_name] = subpackage
                        
                        # Recursively import submodules
                        sub_results = import_submodules_with_resources(subpackage_name)
                        results.update(sub_results)
                    except ImportError as e:
                        print(f"Failed to import {subpackage_name}: {e}")
                        
    except ModuleNotFoundError as e:
        print(f"Package {package_name} not found: {e}")
        
    return results

# Example usage
modules = import_submodules_with_resources('urllib')
print("Imported modules:", list(modules.keys())[:3])  # Show first 3

Imported modules: ['urllib.parse', 'urllib.request', 'urllib.error']

Comparison

Key Points

• pkgutil.iter_modules() is the traditional approach that works across all Python versions
• importlib.resources provides better resource handling for modern Python versions
• Both methods handle recursive importing of subpackages
• Error handling is important when importing unknown modules

Conclusion

Use pkgutil.iter_modules() for broad compatibility and simple cases. Use importlib.resources for Python 3.9+ when you need advanced resource handling capabilities.