Profiling in Python

Python profiling is the process of measuring performance of different parts of a program to identify optimization areas and bottlenecks. Python provides several built-in modules and third-party tools for profiling code execution time, memory usage, and function calls.

Using cProfile for Function Profiling

Function profiling measures execution time of individual functions in your program. Python's built-in cProfile module is the most common tool for this purpose.

Basic Function Profiling

The cProfile.run() function executes code and provides detailed statistics

import cProfile

def calculate_sum(n):
    total = 0
    for i in range(n):
        total += i
    return total

def display_result():
    result = calculate_sum(1000)
    print(f"Sum: {result}")

# Profile the function
cProfile.run('display_result()')

Sum: 499500
         4 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 <stdin>:1(calculate_sum)
        1    0.000    0.000    0.000    0.000 <stdin>:6(display_result)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Profiling with pstats

For better control over profiling output, use pstats module

import cProfile
import pstats
import io

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

# Create profiler and run code
profiler = cProfile.Profile()
profiler.enable()

result = fibonacci(20)
print(f"Fibonacci(20): {result}")

profiler.disable()

# Create stats object and sort by cumulative time
stats = pstats.Stats(profiler)
stats.sort_stats('cumulative')
stats.print_stats(5)  # Show top 5 functions

Fibonacci(20): 6765
         21891 function calls (21 primitive calls) in 0.008 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  21891/1    0.008    0.000    0.008    0.008 <stdin>:1(fibonacci)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Using Line Profiling

Line profiling measures execution time of each individual line of code. This requires the line_profiler package

Installation and Usage

First install the line profiler

pip install line_profiler

Create a script with the @profile decorator

# Save as example.py
@profile
def process_data():
    data = []
    for i in range(1000):
        data.append(i ** 2)
    
    result = sum(data)
    return result

if __name__ == "__main__":
    output = process_data()
    print(f"Result: {output}")

Run the profiler from command line

kernprof -l -v example.py

Using Memory Profiling

Memory profiling tracks memory usage of your program over time. The memory_profiler package provides this functionality.

Installation and Basic Usage

Install the memory profiler

pip install memory_profiler

Use the @profile decorator to monitor memory usage

# Save as memory_example.py
from memory_profiler import profile

@profile
def create_large_list():
    # Create a large list
    large_list = [i for i in range(100000)]
    
    # Process the list
    squared = [x ** 2 for x in large_list]
    
    # Calculate sum
    total = sum(squared)
    return total

if __name__ == "__main__":
    result = create_large_list()
    print(f"Total: {result}")

Run with memory profiling

python -m memory_profiler memory_example.py

Comparison of Profiling Methods

Conclusion

Use cProfile for general function profiling, line_profiler for detailed line analysis, and memory_profiler for memory usage tracking. Choose the appropriate tool based on whether you need to optimize execution time or memory consumption.