How do I parcel out work among a bunch of worker threads in Python?

To parcel out work among a bunch of worker threads in Python, use the concurrent.futures module, especially the ThreadPoolExecutor class. Alternatively, if you want fine control over the dispatching algorithm, you can write your own logic manually using the queue module.

The Queue class maintains a list of objects and has a .put(obj) method that adds items to the queue and a .get() method to return them. The class will take care of the locking necessary to ensure that each job is handed out exactly once.

Using ThreadPoolExecutor

The simplest approach is using ThreadPoolExecutor ?

import concurrent.futures
import time

def process_task(task_id):
    print(f"Processing task {task_id}")
    time.sleep(0.5)  # Simulate work
    return f"Task {task_id} completed"

# Create a pool of 3 worker threads
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    # Submit tasks
    tasks = [executor.submit(process_task, i) for i in range(5)]
    
    # Get results
    for future in concurrent.futures.as_completed(tasks):
        result = future.result()
        print(result)

Processing task 0
Processing task 1
Processing task 2
Processing task 3
Processing task 4
Task 2 completed
Task 0 completed
Task 1 completed
Task 3 completed
Task 4 completed

Manual Queue-Based Approach

For fine-grained control, use a queue to distribute work manually ?

import threading, queue, time

# The worker thread gets jobs off the queue
def worker():
    print('Running worker')
    time.sleep(0.1)
    while True:
        try:
            arg = q.get(block=False)
        except queue.Empty:
            print('Worker', threading.current_thread().name, 'queue empty')
            break
        else:
            print('Worker', threading.current_thread().name, 'running with argument', arg)
            time.sleep(0.2)  # Simulate work

# Create a queue
q = queue.Queue()

# Start a pool of 3 workers
threads = []
for i in range(3):
    t = threading.Thread(target=worker, name=f'worker-{i+1}')
    t.start()
    threads.append(t)

# Add work to the queue
for i in range(10):
    q.put(i)

# Wait for all threads to complete
for t in threads:
    t.join()

print('All work completed')

Running worker
Running worker
Running worker
Worker worker-1 running with argument 0
Worker worker-2 running with argument 1
Worker worker-3 running with argument 2
Worker worker-1 running with argument 3
Worker worker-2 running with argument 4
Worker worker-3 running with argument 5
Worker worker-1 running with argument 6
Worker worker-2 running with argument 7
Worker worker-3 running with argument 8
Worker worker-1 running with argument 9
Worker worker-2 queue empty
Worker worker-3 queue empty
Worker worker-1 queue empty
All work completed

Comparison

Conclusion

Use ThreadPoolExecutor for simple parallel task execution. Use manual queue approach when you need fine control over work distribution and thread behavior.