Posted on by & filed under Content - Highlights and Reviews, programming, Programming & Development, python, Tech.

By Daniel Arbuckle

Daniel Arbuckle is a computer scientist and software engineer who has worked with, studied, and built a career around Python starting with version 1.5. He is the author of multiple books and videos on Python for Packt Publishing.

Mastering Python  Python Testing  Learning Python Testing


Python’s first steps toward an implementation of coroutines came in December 2001 with the release of Python 2.2. At that time, generators were added to the language and Python 2.5 made it easier to use generators as coroutines by changing the yield statement into an expression which also made it easier to send information into a generator as the result of a yield expression.

Now, Python 3.3 made it even easier to use generators as coroutines thanks to a new yield from syntax.

As of Python 3.4, the asyncio framework is part of the standard library and generator-based coroutines were part of the toolkit. Asynchronous execution in a single thread is great for I/O scheduling and lightweight threading. Now, asyncio gives us all of that in one convenient package.

All of which brings us to the recent release of Python 3.5, which includes an explicit coroutine type, distinct from generators, new asynchronous async def, async for, and async with statements, and an await expression that takes the place of yield from for coroutines.

Why does Python need explicit coroutines and new syntax if generator-based coroutines had become strong enough for inclusion in the standard library? The short answer is that generators are primarily for iteration, so using them for something else (no matter how well it works conceptually) introduces ambiguities. For example, if you hand a generator to Python’s for loop, it’s not going to treat it as a coroutine but instead as an iterable.

There were ways to work around this, of course, but it required some odd idioms:

That works, but it’s a bit strange. Worse, that’s a best case. The idiom gets even stranger when it’s not known ahead of time how many objects will need to be iterated over.
It’s much clearer now in Python 3.5:

There’s another problem, related to Python’s special protocol methods such as __enter__ and __exit__ which are called by the code in the Python interpreter, leaving the programmer with no opportunity to yield from it. That meant that generator-based coroutines were not compatible with various important bits of Python syntax such as the with statement. A coroutine couldn’t be called from anything that was called by a special method, whether directly or indirectly, nor was it possible to wait on a future value in a function that was.

The new changes to Python are meant to address these problems. So, what exactly are these changes?

async def is used to define a coroutine. Apart from the asynckeyword, the syntax is almost identical to a normal def statement.

The big differences are, first, that coroutines can contain await, async for, and async with syntaxes, and, second, they are not generators, so they’re not allowed to contain yield or yield from expressions. It’s impossible for a single function to be both a generator and a coroutine.

await is used to pause the current coroutine until the requested coroutine returns. In other words, an await expression is just like a function call, except that the called function can pause, allowing the scheduler to run some other thread for a while. If you try to use an awaitexpression outside of an async def, Python will raise a syntax error.

async with is used to interface with an asynchronous context manager, which is just like a normal context manager except that instead of __enter__ and __exit__ methods, it has __aenter__ and __aexit__ coroutine methods. Because they’re coroutines, these methods can do things like wait for data to come in over the network, without blocking the whole program.

async for is used to get data from an asynchronous iterable.

Asynchronous iterables have an __aiter__ coroutine method which functions like the normal __iter__ method, but can participate in coroutine scheduling and asynchronous I/O. __aiter__ should return an object with an __anext__ coroutine method, which can participate in coroutine scheduling and asynchronous I/O before returning the next iterated value or raising StopAsyncIteration.

All of these new features and the convenience they represent are 100% compatible with the existing asyncio scheduler.

Further, as long as you used the @asyncio.coroutine decorator, your existing asyncio code is also forward compatible with these features without any overhead.

Keep Learning Python

My new video series called “Mastering Python” is available in full on Safari and I have written two books on Python testing called “Learning Python Testing” and “Python Testing Beginner’s Guide.”

Tags: asynchronous, coroutine, Python, syntax,

Comments are closed.