The Python multiprocessing library is a native library that allows virtual parallelization of processes in Python. It’s pretty easy to run multiple functions in parallel in Python, but it’s more complicated to do it when those functions have arguments. In this post we’re going to cover:
- What Python Multiprocessing Processes Are
- Creating Functions to Run in Parallel
- How to Pass Arguments to Multiprocessing Processes in Python
- How to Pass Keyword Arguments to Multiprocessing Processes
- Summary of Using Functions with Arguments for Multiprocessing
Python Multiprocessing Processes
The Python multiprocessing
library allows for process-based parallelism. The Process
class objects represent activities running in separate processes. A process is an instance of a computer function that is running on one or multiple threads. Processes are the abstraction between programs and threads.
Creating Functions to Run in Parallel
For this example, the only thing we’ll need to import is the Process
class from the multiprocessing
library. We’ll create two functions that each take one parameter or argument to run in parallel. Both functions will do the same thing. They will take the counter and increment it by 1 100 times. In each of the 100 increments, the function will print out the function name and the counter value. At the end of each function, we’ll print out that the function has ended.
from multiprocessing import Process
def func1(counter: int):
print("start func 1")
for i in range(100):
counter += 1
print("func 1", counter)
print("end func 1")
def func2(counter: int):
print("start func 2")
for i in range(100):
counter += 1
print("func 2", counter)
print("end func 2")
How to Pass Arguments to Multiprocessing Processes in Python
Now that we’ve created two functions that each take one parameter to run in parallel, let’s run them in parallel. We’ll need to create two variables to pass as arguments. Then we’ll create the Process
objects themselves. This is where we pass in the arguments. We’ll pass in the function name as the target
and a tuple with the counter variable as the args
. Notice that we’ve added a trailing comma to the counter in the arguments, this is to ensure that the args
are interpreted correctly.
if __name__ == "__main__":
counter1 = 0
counter2 = 0
p1 = Process(target = func1, args=(counter1,))
p2 = Process(target = func2, args=(counter2,))
p1.start()
p2.start()
p1.join()
p2.join()
Passing Keyword Arguments to Multiprocessing Processes
Passing a tuple of arguments isn’t the only way we can pass arguments into the Process
class. We can also pass in arguments corresponding to the parameter name using the kwargs
parameter in the Process
class. Instead of passing a tuple, we pass a dictionary to kwargs
where we specify the argument name and the variable being passed in as that argument.
if __name__ == "__main__":
counter1 = 0
counter2 = 0
p1 = Process(target = func1, kwargs={"counter":counter1})
p2 = Process(target = func2, kwargs={"counter":counter2})
p1.start()
p2.start()
p1.join()
p2.join()
Summary of Using Functions with Arguments for Multiprocessing
In this post we briefly discussed what a Process was and how it works with multiprocessing in Python. Then, we went over an example where we ran two functions with arguments in parallel. First we showed how to pass arguments to a function using args
, then with kwargs
. These are the two ways we can do multiprocessing in Python with arguments.
Further Reading
- Matrix Multiplication in Python
- Runtime Error: Event Loop is Closed asyncio Fix
- Run Multiple Functions in Parallel in Python 3
- Build a GRU RNN in Keras
- Send API Requests Asynchronously in Python
I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.
Make a one-time donation
Make a monthly donation
Make a yearly donation
Choose an amount
Or enter a custom amount
Your contribution is appreciated.
Your contribution is appreciated.
Your contribution is appreciated.
DonateDonate monthlyDonate yearly
4 thoughts on “Python Multiprocessing with Arguments”