Python Multiprocessing Functions with Dependencies

Python multiprocessing is used for virtually running programs in parallel. We can use multiprocessing to simply run functions in parallel and run functions that need arguments in parallel. Sometimes, we also need to be able to access the values generated or changed by the functions we run. In this post we’re going to cover how to run multiprocessing functions with dependencies on each other. We’ll go over:

• Python Multiprocessing Process Return Values
• Passing Variables to Processes
• Sharing Memory State Between Python Processes
• Changing the Same Variable while Multiprocessing in Python
• Summary of Python Multiprocessing Functions with Dependencies

Python Multiprocessing Process Return Values

You can put a return statement in the functions that you run in a Process object but running the process will not get you access to the desired return variable. Setting a variable equal to the Process task will get you a Process object. 

The object can call start() and join() to start and wait for the process to end respectively. However, none of these will return the return value. Since we can’t directly access the return value of a function run through multiprocessing we have to access the variables we want to change through sharing memory or state between processes.

Passing Variables to Processes

We can pass variables to a function running in a Process during the instantiation of the Process. When we initialize a Process object, we pass the function in as the target. Two other parameters that a Process object can take are args and kwargs. We can pass in a tuple of arguments to args and a dictionary of parameter names as keys with variable names as values passed in to kwargs.

For a more detailed guide see How to do Python Multiprocessing with Arguments. 

Sharing Memory State Between Python Processes

The key to using parallel functions on the same variable is to share the memory state between processes. This is generally not recommended unless you know what you’re doing. However, it is the simplest way to work with multiple Process objects with dependencies on each other. For this example, we’re going to use a Value object from the multiprocessing library. 

The Value object is an object of type sharedctypes.Synchronized from the multiprocessing library. It is a shared class type that “synchronizes” between processes. These objects are the only objects that can be shared between Process objects to track value changes in multiple processes.

Creating Dependent Functions to Run in Parallel

As always, we’ll begin our program with importing libraries we need. We need the Process and Value class from multiprocessing. We’ll create two functions that increment a counter by one. Each of these functions will print out the value of the counter each time it’s incremented. Then we’ll create two functions that each run both of those functions in parallel.

from multiprocessing import Process, Value
 
def func1(counter):
    print("start func 1")
    for i in range(100):
        counter.value += 1
        print("func 1", counter.value)
    print("end func 1")
 
def func2(counter):
    print("start func 2")
    for i in range(100):
        counter.value += 1
        print("func 2", counter.value)
    print("end func 2")
 
def func3(counter):
    p1 = Process(target=func1, args=(counter,))
    p2 = Process(target=func2, args=(counter,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(f"func3, {counter.value}")
 
def func4(counter):
    p1 = Process(target=func1, args=(counter,))
    p2 = Process(target=func2, args=(counter,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(f"func4, {counter.value}")

Running Functions on the Same Variable in Parallel

Now that we’ve created two functions that run two other functions in parallel, let’s run those functions in parallel. This is where the magic happens, we have to pass in the counter variable here. 

We’ll pass in the counter variable, which is of the Value type from the multiprocessing library to each process. Then we’ll run the processes and print out the value of counter at the end. At the end, our variable should have a value of 400 as shown in the picture at the bottom.

if __name__ =="__main__":
    counter = Value('d', 0)
    p1 = Process(target=func3, args=(counter,))
    p2 = Process(target=func4, args=(counter,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(f"main, {counter.value}")

One thing to be aware of is that sharing memory across multiple functions is not thread safe. That means you may get slightly different results each time. The functions will not end at an increment of 100 because they all affect each other.

Summary of Python Multiprocessing Functions with Dependencies

In this post we learned how we can work with multiprocessing processes that operate on the same variables. We learned that the only way we can affect the same variables is through shared memory states. Then we showed how we can run two functions in parallel that run two other functions in parallel.

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.