Tuesday, February 20, 2018

Introduction to Multiprocessing in Python

Introduction to Multiprocessing in Python

The multiprocessing package supports spawning processes using an API similar to the threading module. It also offers both local and remote concurrency. This tutorial will discuss multiprocessing in Python and how to use multiprocessing to communicate between processes and perform synchronization between processes, as well as logging.

Introduction to Multiprocessing

Multiprocessing works by creating a Process object and then calling its start() method as shown below.

In the example code above, we first import the Process class and then instantiate the Process object with the greeting function which we want to run.

We then tell the process to begin using the start() method, and we finally complete the process with the join() method. 

Additionally, you can also pass arguments to the function by providing the args keyword argument like so:

Example

Let's look at a more detailed example that covers all the concepts we have discussed above.

In this example, we are going to create a process that calculates the square of numbers and prints the results to the console.

You can also create more than one process at the same time, as shown in the example below, in which process p1 gets the results of numbers squared, while the second process p2 checks if the given numbers are even.

Communication Between Processes

Multiprocessing supports two types of communication channels between processes:

  • Pipes
  • Queues

Queues

Queue objects are used to pass data between processes. They can store any pickle-able Python object, and you can use them as shown in the example below:

In the above example, we first create a function that checks if a number is even and then put the result at the end of the queue. We then instantiate a queue object and a process object and begin the process.

Finally, we check if the queue is empty, and if not, we get the values from the front of the queue and print them to the console.

We have shown how to share data between two processes using a queue, and the result is as shown below.

It's also important to note that Python has a Queue module which lives in the process module and is used to share data between threads, unlike the multiprocessing queue which lives in shared memory and is used to share data between processes.

Pipes

Pipes in multiprocessing are primarily used for communication between processes. Usage is as simple as:

Pipe() returns two connection objects which represent the two ends of the pipe. Each connection object has send() and recv() methods. Here we create a process that prints the string hello world and then shares the data across.

Result

Locks

Locks work by ensuring that only one process is executed at a time, hence blocking other processes from executing similar code. This allows the process to be completed, and only then can the lock be released.

The example below shows a pretty straightforward usage of the Lock method.

In this code, we first import the Lock method, acquire it, execute the print function, and then release it.

Logging

The multiprocessing module also provides support for logging, although the logging package doesn't use locks so messages between processes might end up being mixed up during execution.

Usage of logging is as simple as:

Here we first import the logging and multiprocessing modules, and we then define the multiprocessing.log_to_stderr() method, which performs a call to get_logger() as well as adding a handler which sends output to sys.stderr. Finally, we set the logger level and the message we want to convey.

Conclusion

This tutorial has covered what is necessary to get started with multiprocessing in Python. Multiprocessing overcomes the problem of GIL (Global Interpreter Lock) since it leverages the use of subprocesses instead of threads.

There is much more in the Python documentation that isn’t covered in this tutorial, so feel free to visit the Python multiprocessing docs and utilize the full power of this module.


No comments:

Post a Comment