Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
Synchronization and Pooling of processes in Python
Synchronization between processes
Multiprocessing is a package which supports spawning processes using an API. This package is used for both local and remote concurrencies. Using this module, programmer can use multiple processors on a given machine. It runs on Windows and UNIX os.
All equivalent synchronization primitives are present in this package.
Example code
from multiprocessing import Process, Lock
def my_function(x, y):
x.acquire()
print ('hello world', y)
x.release()
if __name__ == '__main__':
lock = Lock()
for num in range(10):
Process(target= my_function, args=(lock, num)).start()Here one instance can lock to ensure that only one process can display standard output at a time.
Pooling
For pooling, we use Pool class. When one can create a pool of processes that will carry the whole tasks submitted to it.
class multiprocessing.Pool([processes[, initializer[, initargs[, maxtasksperchild]]]])
A pool object controls a pool of worker to select which job can be submitted and it supports asynchronous result which has timeouts, callbacks and a parallel map implementation.
cpu_count() is used if the process is none and initializer(*initargs) this function calls when initializer is not none.
apply(func[, args[, kwds]])
This is same as apply() built-in function. This is blocked until the result is ready, if it wants to perform in parallel then apply_async() method is better.
apply_async(func[, args[, kwds[, callback]]])
Returns a result object.
map(func, iterable [, chunksize])
map() is a built-in function and it supports only one iterable argument. It blocks until the result is ready.
In this method, the iterable breaks into a number of small chunks and these small parts are submitted to the process pool as a separate task.
map_async(func, iterable[, chunksize[, callback]])
Returns a result object.
imap(func, iterable[, chunksize])
It is same as itertools.imap().
The size of the argument is same as the one used in map().
imap_unordered(func, iterable[, chunksize])
This is same as imap() except that the retuning iterator should be ordered.
close()
When a worker has been completed all the tasks then worker exits the process.
terminate()
If we want to stop worker process immediately without completing the task then this method is used.
join()
Before using join() method, we must use close() and terminate() functions.
class multiprocessing.pool.AsyncResult
Returned by Pool.apply_async() and Pool.map_async().
get([timeout])
This function returns the result when it arrives.
wait([timeout])
Using this wait function, we wait the result to be available or until timeout seconds pass.
ready()
This function returns whether the call has completed or not.
successful()
This function returns when the call completed without any error.
Example code
# -*- coding: utf-8 -*- """ Created on Sun Sep 30 12:17:58 2018 @author: Tutorials Point """ from multiprocessing import Pool import time def myfunction(m): return m*m if __name__ == '__main__': my_pool = Pool(processes=4) # start 4 worker processes result = my_pool.apply_async(myfunction, (10,)) # evaluate "f(10)" asynchronously in a single process print (result.get(timeout=1)) print (my_pool.map(myfunction, range(10))) # prints "[0, 1, 4,..., 81]" my_it = my_pool.imap(myfunction, range(10)) print (my_it.next() ) # prints "0" print (my_it.next() ) # prints "1" print (my_it.next(timeout=1) ) # prints "4" unless your computer is *very* slow result = my_pool.apply_async(time.sleep, (10,)) print (result.get(timeout=1) ) # raises multiprocessing.TimeoutError
539 Views
