One of Python’s main weaknesses is its inability to have true parallelization due to the Global Interpreter Lock. However, some functions can still virtually run in parallel. Python allows this with two different concepts: multithreading and multiprocessing. In this post we’ll go over how to run two functions virtually in parallel with multiprocessing.
Overview of Running Multiple Functions in Parallel in Python 3
In this post we’ll cover:
- Python Multiprocessing Capabilities
- Using Multiprocessing in Python
- How to Run Multiple Functions in Parallel in Python
- Example Functions That Can Run at the Same Time
- Executing Multiple Functions at the Same Time
Python Multiprocessing Capabilities
Multiprocessing is a native Python library that supports process based parallelism. The multiprocessing library essentially sidesteps the problem of the Global Interpreter Lock by using processes instead of threads. The GIL stops Python from multithreading, but processes are slightly different.
Using Multiprocessing in Python
multiprocessing library in Python allows a user to leverage multiple processors on the same machine. It works on both Windows and Unix based systems. The
multiprocessing library’s APIs are mostly analogous to Python’s threading APIs. The most basic class in the
threading class is
Thread and the most basic class in the
multiprocessing library is
How to Run Multiple Functions at the Same Time in Python3
Just like we execute
Threads using the
join() functions, we also execute
join() APIs. An important aspect of using the
multiprocessing library in Python is that you have to run the processes under
if __name__ == “__main__”. Why? Because new subprocesses are Python interpreters that are started from scratch. They need to be able to find the main module in order to import all the necessary libraries.
Example Functions That Can Run in Parallel
We’ll define two example functions that can run in parallel. Functions that do not rely on each other are functions that can be run in parallel. In our example, we’ll create two simple functions that count from 0 to 100. Both functions will start by setting a counter to 0 and printing out that the function has started. Then each function will increment the counter 100 times and print out the function and the counter value. At the end of each function, they will print out that the function has ended.
def func1(): counter = 0 print("start func 1") while counter < 100: counter += 1 print("func 1", counter) print("end func 1") def func2(): counter = 0 print("start func 2") while counter < 100: counter += 1 print("func 2", counter) print("end func 2")
Executing Multiple Functions at the Same Time
As we said earlier, we need to use the
if __name__ == “__main__” pattern to successfully run the functions in parallel. In our main function we’ll start the two functions we made earlier as
Process‘s. We pass in the function to the
Process through the
target keyword parameter. After defining each
start() and then
join() our processes.
if __name__ == "__main__": p1 = Process(target = func1) p2 = Process(target = func2) p1.start() p2.start() p1.join() p2.join()
We should see an output like the one below.
Summary of Running Multiple Functions in Parallel in Python 3
In this post we learned about running multiple functions in parallel in Python3 using processes through the
multiprocessing library. We briefly covered why Python cannot achieve true parallelism through threading because of the Global Interpreter Lock. We also went over how the
threading libraries provide many similar API structures. Then we went over an example of functions that can run in parallel and how to run them with multiple
- Multiprocessing Overview
- Kruskal’s Algorithm in Python
- Neural Network Code in Python 3 From Scratch
- Dijkstra’s Algorithm in Python‘
- NLP: GRU Model in Python
To learn more, feel free to reach out to me @yujian_tang on Twitter, connect with me on LinkedIn, and join our Discord. Remember to follow the blog to stay updated with cool Python projects and ways to level up your Software and Python skills! If you liked this article, please Tweet it, share it on LinkedIn, or tell your friends!
I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.
Make a one-time donation
Make a monthly donation
Make a yearly donation
Choose an amount
Or enter a custom amount
Your contribution is appreciated.
Your contribution is appreciated.
Your contribution is appreciated.DonateDonate monthlyDonate yearly