Python Multiprocessing with Arguments

The Python multiprocessing library is a native library that allows virtual parallelization of processes in Python. It’s pretty easy to run multiple functions in parallel in Python, but it’s more complicated to do it when those functions have arguments. In this post we’re going to cover:

• What Python Multiprocessing Processes Are
• Creating Functions to Run in Parallel
• How to Pass Arguments to Multiprocessing Processes in Python
• How to Pass Keyword Arguments to Multiprocessing Processes
• Summary of Using Functions with Arguments for Multiprocessing

Python Multiprocessing Processes

The Python multiprocessing library allows for process-based parallelism. The Process class objects represent activities running in separate processes. A process is an instance of a computer function that is running on one or multiple threads. Processes are the abstraction between programs and threads.

Creating Functions to Run in Parallel

For this example, the only thing we’ll need to import is the Process class from the multiprocessing library. We’ll create two functions that each take one parameter or argument to run in parallel. Both functions will do the same thing. They will take the counter and increment it by 1 100 times. In each of the 100 increments, the function will print out the function name and the counter value. At the end of each function, we’ll print out that the function has ended.

from multiprocessing import Process
 
def func1(counter: int):
    print("start func 1")
    for i in range(100):
        counter += 1
        print("func 1", counter)
    print("end func 1")
 
def func2(counter: int):
    print("start func 2")
    for i in range(100):
        counter += 1
        print("func 2", counter)
    print("end func 2")

How to Pass Arguments to Multiprocessing Processes in Python

Now that we’ve created two functions that each take one parameter to run in parallel, let’s run them in parallel. We’ll need to create two variables to pass as arguments. Then we’ll create the Process objects themselves. This is where we pass in the arguments. We’ll pass in the function name as the target and a tuple with the counter variable as the args. Notice that we’ve added a trailing comma to the counter in the arguments, this is to ensure that the args are interpreted correctly.

if __name__ == "__main__":
    counter1 = 0
    counter2 = 0
    p1 = Process(target = func1, args=(counter1,))
    p2 = Process(target = func2, args=(counter2,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Passing Keyword Arguments to Multiprocessing Processes

Passing a tuple of arguments isn’t the only way we can pass arguments into the Process class. We can also pass in arguments corresponding to the parameter name using the kwargs parameter in the Process class. Instead of passing a tuple, we pass a dictionary to kwargs where we specify the argument name and the variable being passed in as that argument.

if __name__ == "__main__":
    counter1 = 0
    counter2 = 0
    p1 = Process(target = func1, kwargs={"counter":counter1})
    p2 = Process(target = func2, kwargs={"counter":counter2})
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Summary of Using Functions with Arguments for Multiprocessing

In this post we briefly discussed what a Process was and how it works with multiprocessing in Python. Then, we went over an example where we ran two functions with arguments in parallel. First we showed how to pass arguments to a function using args, then with kwargs. These are the two ways we can do multiprocessing in Python with arguments.

Further Reading

• Matrix Multiplication in Python
• Runtime Error: Event Loop is Closed asyncio Fix
• Run Multiple Functions in Parallel in Python 3
• Build a GRU RNN in Keras
• Send API Requests Asynchronously in Python

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.