Synchronization and Pooling of processes in Python

Python's multiprocessing module provides tools for synchronization and process pooling to handle concurrent execution. Synchronization ensures processes don't interfere with each other, while pooling manages multiple worker processes efficiently.

Synchronization Between Processes

The multiprocessing package supports spawning processes using an API similar to the threading module. It works on both Windows and UNIX operating systems and provides synchronization primitives to coordinate between processes.

Example

Here's how to use a Lock to synchronize output from multiple processes ?

from multiprocessing import Process, Lock

def my_function(lock, num):
    lock.acquire()
    try:
        print('Process', num, 'is running')
    finally:
        lock.release()

if __name__ == '__main__':
    lock = Lock()
    processes = []
    
    for num in range(5):
        p = Process(target=my_function, args=(lock, num))
        processes.append(p)
        p.start()
    
    for p in processes:
        p.join()

Process 0 is running
Process 1 is running
Process 2 is running
Process 3 is running
Process 4 is running

The Lock ensures that only one process can access the shared resource (stdout) at a time, preventing garbled output.

Process Pooling

The Pool class creates a pool of worker processes to execute tasks in parallel. It's more efficient than creating individual processes for each task.

Pool Class Methods

apply(func[, args[, kwds]]) ? Executes function synchronously (blocks until complete)

apply_async(func[, args[, kwds[, callback]]]) ? Executes function asynchronously and returns a result object

map(func, iterable[, chunksize]) ? Applies function to each item in iterable, similar to built-in map()

map_async(func, iterable[, chunksize[, callback]]) ? Asynchronous version of map()

close() ? Prevents more tasks from being submitted to the pool

terminate() ? Stops worker processes immediately

join() ? Waits for worker processes to exit (must call close() first)

Example

Here's a complete example demonstrating different Pool methods ?

from multiprocessing import Pool
import time

def square_number(n):
    return n * n

def slow_function(seconds):
    time.sleep(seconds)
    return f"Slept for {seconds} seconds"

if __name__ == '__main__':
    # Create a pool with 4 worker processes
    with Pool(processes=4) as pool:
        # Synchronous apply
        result = pool.apply(square_number, (10,))
        print(f"Synchronous result: {result}")
        
        # Asynchronous apply
        async_result = pool.apply_async(square_number, (5,))
        print(f"Async result: {async_result.get()}")
        
        # Map function to multiple values
        numbers = range(6)
        squared = pool.map(square_number, numbers)
        print(f"Squared numbers: {squared}")

Synchronous result: 100
Async result: 25
Squared numbers: [0, 1, 4, 9, 16, 25]

Handling Timeouts

You can set timeouts for operations to avoid blocking indefinitely ?

from multiprocessing import Pool, TimeoutError
import time

def slow_task(duration):
    time.sleep(duration)
    return f"Completed after {duration} seconds"

if __name__ == '__main__':
    with Pool(processes=2) as pool:
        try:
            result = pool.apply_async(slow_task, (3,))
            # Wait only 1 second for result
            output = result.get(timeout=1)
            print(output)
        except TimeoutError:
            print("Task timed out!")

Comparison of Methods

Conclusion

Use Lock for process synchronization and Pool for efficient parallel processing. The with statement ensures proper cleanup of pool resources. Choose synchronous methods for simple cases and asynchronous methods when you need non-blocking execution.