Multithreaded Priority Queue in Python

The Queue module in Python allows you to create thread-safe queue objects for multithreaded applications. A priority queue processes items based on their priority rather than insertion order, making it ideal for task scheduling and resource management.

Queue Methods

The Queue module provides several methods to control queue operations ?

• get() − Removes and returns an item from the queue
• put() − Adds an item to the queue
• qsize() − Returns the number of items currently in the queue
• empty() − Returns True if queue is empty; otherwise, False
• full() − Returns True if queue is full; otherwise, False

Basic Multithreaded Queue Example

Here's how to implement a basic multithreaded queue with worker threads ?

import queue
import threading
import time

exitFlag = 0

class WorkerThread(threading.Thread):
    def __init__(self, threadID, name, q):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.q = q
        
    def run(self):
        print("Starting " + self.name)
        process_data(self.name, self.q)
        print("Exiting " + self.name)

def process_data(threadName, q):
    while not exitFlag:
        queueLock.acquire()
        if not workQueue.empty():
            data = q.get()
            queueLock.release()
            print("%s processing %s" % (threadName, data))
        else:
            queueLock.release()
        time.sleep(1)

# Setup
threadList = ["Thread-1", "Thread-2", "Thread-3"]
nameList = ["One", "Two", "Three", "Four", "Five"]
queueLock = threading.Lock()
workQueue = queue.Queue(10)
threads = []
threadID = 1

# Create new threads
for tName in threadList:
    thread = WorkerThread(threadID, tName, workQueue)
    thread.start()
    threads.append(thread)
    threadID += 1

# Fill the queue
queueLock.acquire()
for word in nameList:
    workQueue.put(word)
queueLock.release()

# Wait for queue to empty
while not workQueue.empty():
    pass

# Notify threads it's time to exit
exitFlag = 1

# Wait for all threads to complete
for t in threads:
    t.join()
    
print("Exiting Main Thread")

Starting Thread-1
Starting Thread-2
Starting Thread-3
Thread-1 processing One
Thread-2 processing Two
Thread-3 processing Three
Thread-1 processing Four
Thread-2 processing Five
Exiting Thread-3
Exiting Thread-1
Exiting Thread-2
Exiting Main Thread

Priority Queue Implementation

For priority-based processing, use PriorityQueue which processes items with lower priority numbers first ?

import queue
import threading
import time

def worker(name, pq):
    while True:
        try:
            priority, item = pq.get(timeout=2)
            print(f"{name} processing priority {priority}: {item}")
            time.sleep(1)
            pq.task_done()
        except queue.Empty:
            print(f"{name} finished - no more tasks")
            break

# Create priority queue
priority_queue = queue.PriorityQueue()

# Add items with priorities (lower number = higher priority)
tasks = [(1, "Critical task"), (3, "Low priority task"), (2, "Medium task")]

for priority, task in tasks:
    priority_queue.put((priority, task))

# Create worker threads
threads = []
for i in range(2):
    t = threading.Thread(target=worker, args=(f"Worker-{i+1}", priority_queue))
    t.start()
    threads.append(t)

# Wait for all tasks to complete
priority_queue.join()

# Wait for threads to finish
for t in threads:
    t.join()
    
print("All tasks completed")

Worker-1 processing priority 1: Critical task
Worker-2 processing priority 2: Medium task
Worker-1 processing priority 3: Low priority task
Worker-1 finished - no more tasks
Worker-2 finished - no more tasks
All tasks completed

Queue Types Comparison

Conclusion

Multithreaded priority queues enable efficient task distribution among worker threads. Use PriorityQueue when tasks have different importance levels, and regular Queue for standard FIFO processing.