Run Multiple Functions in Parallel in Python 3

One of Python’s main weaknesses is its inability to have true parallelization due to the Global Interpreter Lock. However, some functions can still virtually run in parallel. Python allows this with two different concepts: multithreading and multiprocessing. In this post we’ll go over how to run two functions virtually in parallel with multiprocessing.

Overview of Python Parallelism

In this post we’ll cover:

• Python Parallelism and Multiprocessing Capabilities
  • Using Multiprocessing in Python
• How to Run Multiple Functions in Parallel in Python
  • Example Functions That Can Run at the Same Time
  • Executing Multiple Functions at the Same Time

Python Parallelism and Multiprocessing Capabilities

Multiprocessing is a native Python library that supports process based parallelism. The multiprocessing library essentially sidesteps the problem of the Global Interpreter Lock by using processes instead of threads. The GIL stops Python from multithreading, but processes are slightly different.

Using Multiprocessing in Python

Using the multiprocessing library in Python allows a user to leverage multiple processors on the same machine. It works on both Windows and Unix based systems. The multiprocessing library’s APIs are mostly analogous to Python’s threading APIs. The most basic class in the threading class is Thread and the most basic class in the multiprocessing library is Process.

How to Run Multiple Functions at the Same Time in Python 3

Just like we execute Threads using the start() and join() functions, we also execute Process with start() and join() APIs. An important aspect of using the multiprocessing library in Python is that you have to run the processes under if __name__ == “__main__”. Why? Because new subprocesses are Python interpreters that are started from scratch. They need to be able to find the main module in order to import all the necessary libraries.

Example Functions That Can Run in Parallel

We’ll define two example functions that can run in parallel. Functions that do not rely on each other are functions that can be run in parallel. In our example, we’ll create two simple functions that count from 0 to 100. Both functions will start by setting a counter to 0 and printing out that the function has started. Then each function will increment the counter 100 times and print out the function and the counter value. At the end of each function, they will print out that the function has ended.

def func1():
    counter = 0
    print("start func 1")
    while counter < 100:
        counter += 1
        print("func 1", counter)
    print("end func 1")
 
def func2():
    counter = 0
    print("start func 2")
    while counter < 100:
        counter += 1
        print("func 2", counter)
    print("end func 2")

Executing Multiple Functions at the Same Time

As we said earlier, we need to use the if __name__ == “__main__” pattern to successfully run the functions in parallel. In our main function we’ll start the two functions we made earlier as Process‘s. We pass in the function to the Process through the target keyword parameter. After defining each Process we start() and then join() our processes. 

if __name__ == "__main__":
    p1 = Process(target = func1)
    p2 = Process(target = func2)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

We should see an output like the one below.

Summary of How to Start all Processes Simultaneously in Python

In this post we learned about running multiple functions in parallel in Python3 using processes through the multiprocessing library. We briefly covered why Python cannot achieve true parallelism through threading because of the Global Interpreter Lock. We also went over how the multiprocessing and threading libraries provide many similar API structures. Then we went over an example of functions that can run in parallel and how to run them with multiple Process objects.

Further Reading

• Multiprocessing Overview
• Kruskal’s Algorithm in Python
• Neural Network Code in Python 3 From Scratch
• Dijkstra’s Algorithm in Python‘
• NLP: GRU Model in Python

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.