Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
ProcessPoolExecutor Class in Python
The ProcessPoolExecutor class in python is part of the concurrent.futures module, is a high level interface for asynchronously executing functions using the processes and threads. The ProcessPoolExecutor allows us to execute the multiple functions in parallel using the multiple processes, which is particularly useful to CPU bound tasks that benefit in the parallelization process.
Multiprocessing and Multithreading
Before going to see about the ProcessPoolExecutor class we have to know about the Multiprocessing and Multithreading. Multiprocessing and Multithreading are the techniques used to achieve the parallelism process and they are different in the way they manage and create concurrent tasks.
Multiprocessing uses multiple processes to execute tasks in parallel and each process has its own memory space to avoid issues with shared memory and concurrency. It is also a means of communication between processes that can be more complex and expensive, as data must be serialized and de−serialized between processes. Multiprocessing is particularly used in CPU−bound tasks which benefit in performing the parallelization tasks like numerical computations or image processing etc. In python we have a module multiprocessing which allows us to create and manage the processes and the pool class is used to manage the pool of worker processes to execute the functions in parallel.
Multithreading uses multiple threads within a single process to achieve parallelism and each thread shares the same memory space as the main thread, which can simplify communication between threads. This means we need to be careful while accessing shared data to avoid issues with concurrency like race conditions or deadlocks. Multithreading used to perform I/O−bound tasks which benefit in parallelization like network programming or file processing etc. In python we have the module threading to create and manage the threads and the thread class used to create the new thread and execute the function in that thread. The lock class is used to synchronize the access of the shared data between the threads.
Creating the ProcessPoolExecutor
The process of creating a ProcessPoolExecutor is similar to the ThreadPoolExecutor only difference is we have to import the class from the concurrent.futures module. We will use the OS module to get the current task PID which we execute within our pool.
Example
from concurrent.futures import ProcessPoolExecutor
import os
def task():
print("Current executing task PID {}".format(os.getpid()))
if __name__ == '__main__':
result =[]
with ProcessPoolExecutor(max_workers=3) as exe:
result1 = exe.submit(task())
Output
Current executing task PID 6652
Example
from concurrent.futures import ProcessPoolExecutor
import os
values = [10,40,30,4]
def square(x):
print(f'square of {x}:{x*x}')
print("Current executing task PID {}".format(os.getpid()))
if __name__ == '__main__':
result =[]
with ProcessPoolExecutor(max_workers = 5) as exe:
for i in values:
result = exe.submit(square(i))
Output
square of 10:100 Current executing task PID 19276 square of 40:1600 Current executing task PID 19276 square of 30:900 Current executing task PID 19276 square of 4:16 Current executing task PID 19276
45 Views
To Continue Learning Please Login