Python: Journey from Novice to Expert

Sharing data between processes

This is really the most difficult part about multiprocessing, multithreading, and distributed programming - which data to pass along and which data to skip. The theory is really simple, however: whenever possible don't transfer any data, don't share anything, and keep everything local. Essentially the functional programming paradigm, which is why functional programming mixes really well with multiprocessing. In practice, regrettably, this is simply not always possible. The multiprocessing library has several options to share data: Pipe, Namespace, Queue, and a few others. All these options might tempt you to share your data between the processes all the time. This is indeed possible, but the performance impact is, ...