Python Multiprocessing with Arguments

The Python multiprocessing library is a native library that allows virtual parallelization of processes in Python. It’s pretty easy to run multiple functions in parallel in Python, but it’s more complicated to do it when those functions have arguments. In this post we’re going to cover:

  • What Python Multiprocessing Processes Are
  • Creating Functions to Run in Parallel
  • How to Pass Arguments to Multiprocessing Processes in Python
  • How to Pass Keyword Arguments to Multiprocessing Processes
  • Summary of Using Functions with Arguments for Multiprocessing

Python Multiprocessing Processes

The Python multiprocessing library allows for process-based parallelism. The Process class objects represent activities running in separate processes. A process is an instance of a computer function that is running on one or multiple threads. Processes are the abstraction between programs and threads.

Creating Functions to Run in Parallel

For this example, the only thing we’ll need to import is the Process class from the multiprocessing library. We’ll create two functions that each take one parameter or argument to run in parallel. Both functions will do the same thing. They will take the counter and increment it by 1 100 times. In each of the 100 increments, the function will print out the function name and the counter value. At the end of each function, we’ll print out that the function has ended.

from multiprocessing import Process
 
def func1(counter: int):
    print("start func 1")
    for i in range(100):
        counter += 1
        print("func 1", counter)
    print("end func 1")
 
def func2(counter: int):
    print("start func 2")
    for i in range(100):
        counter += 1
        print("func 2", counter)
    print("end func 2")

How to Pass Arguments to Multiprocessing Processes in Python

Now that we’ve created two functions that each take one parameter to run in parallel, let’s run them in parallel. We’ll need to create two variables to pass as arguments. Then we’ll create the Process objects themselves. This is where we pass in the arguments. We’ll pass in the function name as the target and a tuple with the counter variable as the args. Notice that we’ve added a trailing comma to the counter in the arguments, this is to ensure that the args are interpreted correctly.

if __name__ == "__main__":
    counter1 = 0
    counter2 = 0
    p1 = Process(target = func1, args=(counter1,))
    p2 = Process(target = func2, args=(counter2,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Passing Keyword Arguments to Multiprocessing Processes

Passing a tuple of arguments isn’t the only way we can pass arguments into the Process class. We can also pass in arguments corresponding to the parameter name using the kwargs parameter in the Process class. Instead of passing a tuple, we pass a dictionary to kwargs where we specify the argument name and the variable being passed in as that argument.

if __name__ == "__main__":
    counter1 = 0
    counter2 = 0
    p1 = Process(target = func1, kwargs={"counter":counter1})
    p2 = Process(target = func2, kwargs={"counter":counter2})
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Summary of Using Functions with Arguments for Multiprocessing

In this post we briefly discussed what a Process was and how it works with multiprocessing in Python. Then we went over an example where we ran two functions with arguments in parallel. First we showed how to pass arguments to a function using args, then with kwargs. These are the two ways we can do multiprocessing in Python with arguments.

Further Reading

Learn More

To learn more, feel free to reach out to me @yujian_tang on Twitter, connect with me on LinkedIn, and join our Discord. Remember to follow the blog to stay updated with cool Python projects and ways to level up your Software and Python skills! If you liked this article, please Tweet it, share it on LinkedIn, or tell your friends!

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.

Yujian Tang
Yujian Tang

I started my professional software career interning for IBM in high school after winning ACSL two years in a row. I got into AI/ML in college where I published a first author paper to IEEE Big Data. After college I worked on the AutoML infrastructure at Amazon before leaving to work in startups. I believe I create the highest quality software content so that’s what I’m doing now. Drop a comment to let me know!

One-Time
Monthly
Yearly

Make a one-time donation

Make a monthly donation

Make a yearly donation

Choose an amount

$5.00
$15.00
$100.00
$5.00
$15.00
$100.00
$5.00
$15.00
$100.00

Or enter a custom amount

$

Your contribution is appreciated.

Your contribution is appreciated.

Your contribution is appreciated.

DonateDonate monthlyDonate yearly

4 thoughts on “Python Multiprocessing with Arguments

Leave a Reply

%d bloggers like this: