How to implement Multithreaded queue With Python

A multithreaded queue is a powerful pattern for distributing work across multiple threads. Python's queue module provides thread-safe queue implementations that allow multiple threads to safely add and remove tasks.

Understanding Queues

A queue is a First In, First Out (FIFO) data structure. Think of it like a grocery store checkout line ? people enter at one end and exit from the other in the same order they arrived.

Key queue operations:

• enqueue ? adds elements to the end
• dequeue ? removes elements from the beginning
• FIFO ? first element added is first to be removed

Python Queue Methods

Python's queue module provides these essential methods:

• put() ? adds a new element to the queue
• get() ? returns and removes the next element
• qsize() ? returns number of current elements
• empty() ? returns True if queue is empty
• full() ? returns True if queue is full

Creating the Worker Function

First, let's create a function that processes queue items. This function multiplies a number by all integers from 1 to itself:

def print_multiply(x):
    output_value = []
    for i in range(1, x + 1):
        output_value.append(i * x)
    print(f"The multiplication result for {x} is - {output_value}")

# Test the function
print_multiply(5)

The multiplication result for 5 is - [5, 10, 15, 20, 25]

Queue Processing Function

Next, we need a function that continuously processes items from the queue until it's empty:

import queue
import time

def process_queue(my_queue):
    while True:
        try:
            value = my_queue.get(block=False)
        except queue.Empty:
            return
        else:
            print_multiply(value)
            time.sleep(1)  # Simulate processing time

Multithreaded Worker Class

We'll create a thread class that runs the queue processing function:

import threading

class WorkerThread(threading.Thread):
    def __init__(self, name, task_queue):
        threading.Thread.__init__(self)
        self.name = name
        self.task_queue = task_queue

    def run(self):
        print(f"Starting thread - {self.name}")
        process_queue(self.task_queue)
        print(f"Completed thread - {self.name}")

Complete Implementation

Here's the complete multithreaded queue implementation:

import queue
import threading
import time

def print_multiply(x):
    output_value = []
    for i in range(1, x + 1):
        output_value.append(i * x)
    print(f"Thread {threading.current_thread().name}: Result for {x} = {output_value}")

def process_queue(task_queue):
    while True:
        try:
            value = task_queue.get(block=False)
        except queue.Empty:
            return
        else:
            print_multiply(value)
            time.sleep(1)  # Simulate work

class WorkerThread(threading.Thread):
    def __init__(self, name, task_queue):
        threading.Thread.__init__(self)
        self.name = name
        self.task_queue = task_queue

    def run(self):
        print(f"Starting thread - {self.name}")
        process_queue(self.task_queue)
        print(f"Completed thread - {self.name}")

# Create and fill the queue
input_values = [2, 4, 6, 5]
task_queue = queue.Queue()

for value in input_values:
    task_queue.put(value)

# Create and start worker threads
threads = []
for i in range(3):
    thread = WorkerThread(f"Worker-{i+1}", task_queue)
    threads.append(thread)
    thread.start()

# Wait for all threads to complete
for thread in threads:
    thread.join()

print("All tasks completed!")

Starting thread - Worker-1
Starting thread - Worker-2
Starting thread - Worker-3
Thread Worker-1: Result for 2 = [2, 4]
Thread Worker-2: Result for 4 = [4, 8, 12, 16]
Thread Worker-3: Result for 6 = [6, 12, 18, 24, 30, 36]
Thread Worker-1: Result for 5 = [5, 10, 15, 20, 25]
Completed thread - Worker-1
Completed thread - Worker-2
Completed thread - Worker-3
All tasks completed!

How It Works

The multithreaded queue system works as follows:

1. Tasks are added to a thread-safe queue
2. Multiple worker threads compete to get tasks from the queue
3. Each thread processes one task at a time
4. When the queue is empty, threads terminate

Benefits of Multithreaded Queues

• Load balancing ? work is distributed evenly across threads
• Thread safety ? Python's queue handles concurrent access
• Scalability ? easy to adjust number of worker threads
• Fault tolerance ? if one thread fails, others continue

Conclusion

Multithreaded queues provide an efficient way to process tasks concurrently. Python's queue.Queue handles thread synchronization automatically, making it ideal for producer-consumer patterns where multiple threads need to safely share work.