Python Concurrency and Distributed Computing Workshop

[4 days] This one-of-kind workshop, designed for more experienced Python programmers, provides an in-depth examination of concurrent programming idioms and standard library modules. The aim of this workshop is to go far beyond the information found in the manual and to gain a deeper understanding of how different approaches work, their performance characteristics, limitations, and types of real-world problems for which they are suited. Course topics include, but are not limited to the following:

• Interpreter execution model
• Thread programming
• The global interpreter lock
• Subprocesses
• Multiprocessing
• Interprocess communication
• Message queues and message passing
• Coroutines
• Asynchronous and non-blocking I/O
• Distributed computing

In addition to the above, this course is taught exclusively using Python 3. Although all of the topics also apply to earlier Python versions, this is a perfect opportunity to see what's new and try it out.

Do you have to be some kind of freaky expert to attend? Absolutely not--the whole point of the workshop is to learn. You'll walk away from the workshop looking at various Python libraries and frameworks in a whole new light. Plus, the whole topic is just cool.

Workshop Agenda

Here is a general outline for the workshop.

1. Introduction. A brief introduction to concurrency and distributed computing topics.
2. Thread Programming. Everything you wanted to know about Python threads, but were afraid to ask. Includes the absolute basics of using the threading module and effective ways to use threads in a way that won't make your code (or your head) explode. Also includes detailed information on the Global Interpreter Lock (GIL), tuning parameters, and the interaction between threads and C/C++ extension modules.
3. Message Passing and Data Serialization. Message passing is a core component of distributed computation. This section provides an in-depth look at different interprocess communication mechanisms, their performance characteristics, and tuning options. Topics include the subprocess module, named pipes, network sockets, memory mapped regions, pickle, marshal, structure packing, and binary I/O. New for 2011: we'll also talk about using Python with 0MQ.
4. Multiprocessing. A tour of features provide by the multiprocessing library that allow you to take advantage of multiple CPU cores. Topics will primarily focus on multiprocessing queues and process pools.

5. Distributed Programming. An in-depth tour of different distributed programming techniques. Topics include programming with actors, client-server computing, REST, remote procedure call, and distributed objects.

6. Advanced I/O handling. A look at different I/O handling techniques including blocking, non-blocking, asynchronous, and event-driven I/O. The primary goal of this section is to better understand the I/O handling using by different libraries and frameworks such as asyncore, Twisted, etc.
7. Generators and Coroutines. An overview of concurrent programming using generators and coroutines. The major focus of this section is on using generators to implement user-level task switching and to better understand libraries based on microthreads, tasklets, green-threads, and similarly named entities.

Format

The workshop consists of highly detailed presentation slides and nearly 50 hands on exercises where you will get to experiment--approximately half the time will be spent working on programs. For a sample of presentation materials, see my talk on the Python GIL. If you are familiar with some of my past PyCON tutorials, you should know that the concurrency workshop has almost no overlap. With the exception of the GIL talk, almost all of the material has not been presented in any other venue.