Run Multiple Functions in Parallel in Python3

One of Python’s main weaknesses is its inability to have true parallelization due to the Global Interpreter Lock. However, some functions can still virtually run in parallel. Python allows this with two different concepts: multithreading and multiprocessing. In this post we’ll go over how to run two functions virtually in parallel with multiprocessing.

Overview of Running Multiple Functions in Parallel in Python 3

In this post we’ll cover:

  • Python Multiprocessing Capabilities
    • Using Multiprocessing in Python
  • How to Run Multiple Functions in Parallel in Python
    • Example Functions That Can Run at the Same Time
    • Executing Multiple Functions at the Same Time

Python Multiprocessing Capabilities

Multiprocessing is a native Python library that supports process based parallelism. The multiprocessing library essentially sidesteps the problem of the Global Interpreter Lock by using processes instead of threads. The GIL stops Python from multithreading, but processes are slightly different.

Using Multiprocessing in Python

Using the multiprocessing library in Python allows a user to leverage multiple processors on the same machine. It works on both Windows and Unix based systems. The multiprocessing library’s APIs are mostly analogous to Python’s threading APIs. The most basic class in the threading class is Thread and the most basic class in the multiprocessing library is Process.

How to Run Multiple Functions at the Same Time in Python3

Just like we execute Threads using the start() and join() functions, we also execute Process with start() and join() APIs. An important aspect of using the multiprocessing library in Python is that you have to run the processes under if __name__ == “__main__”. Why? Because new subprocesses are Python interpreters that are started from scratch. They need to be able to find the main module in order to import all the necessary libraries.

Example Functions That Can Run in Parallel

We’ll define two example functions that can run in parallel. Functions that do not rely on each other are functions that can be run in parallel. In our example, we’ll create two simple functions that count from 0 to 100. Both functions will start by setting a counter to 0 and printing out that the function has started. Then each function will increment the counter 100 times and print out the function and the counter value. At the end of each function, they will print out that the function has ended.

def func1():
    counter = 0
    print("start func 1")
    while counter < 100:
        counter += 1
        print("func 1", counter)
    print("end func 1")
 
def func2():
    counter = 0
    print("start func 2")
    while counter < 100:
        counter += 1
        print("func 2", counter)
    print("end func 2")

Executing Multiple Functions at the Same Time

As we said earlier, we need to use the if __name__ == “__main__” pattern to successfully run the functions in parallel. In our main function we’ll start the two functions we made earlier as Process‘s. We pass in the function to the Process through the target keyword parameter. After defining each Process we start() and then join() our processes. 

if __name__ == "__main__":
    p1 = Process(target = func1)
    p2 = Process(target = func2)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

We should see an output like the one below.

end of example result from running python functions in parallel
Multiprocessing two counter functions in Python

Summary of Running Multiple Functions in Parallel in Python 3

In this post we learned about running multiple functions in parallel in Python3 using processes through the multiprocessing library. We briefly covered why Python cannot achieve true parallelism through threading because of the Global Interpreter Lock. We also went over how the multiprocessing and threading libraries provide many similar API structures. Then we went over an example of functions that can run in parallel and how to run them with multiple Process objects.

Further Reading

Learn More

To learn more, feel free to reach out to me @yujian_tang on Twitter, connect with me on LinkedIn, and join our Discord. Remember to follow the blog to stay updated with cool Python projects and ways to level up your Software and Python skills! If you liked this article, please Tweet it, share it on LinkedIn, or tell your friends!

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.

Yujian Tang
Yujian Tang

I started my professional software career interning for IBM in high school after winning ACSL two years in a row. I got into AI/ML in college where I published a first author paper to IEEE Big Data. After college I worked on the AutoML infrastructure at Amazon before leaving to work in startups. I believe I create the highest quality software content so that’s what I’m doing now. Drop a comment to let me know!

One-Time
Monthly
Yearly

Make a one-time donation

Make a monthly donation

Make a yearly donation

Choose an amount

$5.00
$15.00
$100.00
$5.00
$15.00
$100.00
$5.00
$15.00
$100.00

Or enter a custom amount

$

Your contribution is appreciated.

Your contribution is appreciated.

Your contribution is appreciated.

DonateDonate monthlyDonate yearly

8 thoughts on “Run Multiple Functions in Parallel in Python3

Leave a Reply

%d bloggers like this: