Launching parallel tasks in Python

Parallel processing allows Python programs to break work into independent subprograms that can run simultaneously. This improves performance by utilizing multiple CPU cores and reducing overall execution time.

Using multiprocessing Module

The multiprocessing module creates child processes that run independently from the main process. Each process has its own memory space and can execute code in parallel ?

Basic Process Creation

import multiprocessing
import time

class WorkerProcess(multiprocessing.Process):
    def __init__(self, process_id):
        super(WorkerProcess, self).__init__()
        self.process_id = process_id

    def run(self):
        time.sleep(1)
        print("Running process id: {}".format(self.process_id))

if __name__ == '__main__':
    # Create and run processes sequentially
    for pid in ["a", "b", "c"]:
        p = WorkerProcess(pid)
        p.start()
        p.join()  # Wait for process to complete

Running process id: a
Running process id: b
Running process id: c

True Parallel Execution

To run processes truly in parallel, start all processes first, then join them ?

import multiprocessing
import time

class WorkerProcess(multiprocessing.Process):
    def __init__(self, process_id):
        super(WorkerProcess, self).__init__()
        self.process_id = process_id

    def run(self):
        time.sleep(2)
        print("Process {} completed".format(self.process_id))

if __name__ == '__main__':
    processes = []
    
    # Start all processes
    for pid in ["X", "Y", "Z"]:
        p = WorkerProcess(pid)
        p.start()
        processes.append(p)
    
    # Wait for all processes to complete
    for p in processes:
        p.join()
    
    print("All processes finished")

Process X completed
Process Y completed  
Process Z completed
All processes finished

Using Process Pool

For simpler parallel tasks, use Pool to manage multiple worker processes automatically ?

import multiprocessing
import time

def worker_task(task_id):
    time.sleep(1)
    return "Task {} completed by process {}".format(task_id, multiprocessing.current_process().pid)

if __name__ == '__main__':
    with multiprocessing.Pool(processes=3) as pool:
        tasks = [1, 2, 3, 4, 5]
        results = pool.map(worker_task, tasks)
        
        for result in results:
            print(result)

Task 1 completed by process 12345
Task 2 completed by process 12346
Task 3 completed by process 12347
Task 4 completed by process 12345
Task 5 completed by process 12346

Key Considerations

Conclusion

Use multiprocessing for CPU-intensive tasks that can run independently. Process pools are ideal for simple parallel functions, while custom process classes offer more control for complex workflows.