By Daniel Arbuckle
Daniel Arbuckle is a computer scientist and software engineer who has worked with, studied, and built a career around Python starting with version 1.5. He is the author of multiple books and videos on Python for Packt Publishing.
Python’s first steps toward an implementation of coroutines came in December 2001 with the release of Python 2.2. At that time, generators were added to the language and Python 2.5 made it easier to use generators as coroutines by changing the
yield statement into an expression which also made it easier to send information into a generator as the result of a
Now, Python 3.3 made it even easier to use generators as coroutines thanks to a new
yield from syntax.
As of Python 3.4, the
asyncio framework is part of the standard library and generator-based coroutines were part of the toolkit. Asynchronous execution in a single thread is great for I/O scheduling and lightweight threading. Now,
asyncio gives us all of that in one convenient package.
All of which brings us to the recent release of Python 3.5, which includes an explicit coroutine type, distinct from generators, new asynchronous
async for, and
async with statements, and an
await expression that takes the place of
yield from for coroutines.
Why does Python need explicit coroutines and new syntax if generator-based coroutines had become strong enough for inclusion in the standard library? The short answer is that generators are primarily for iteration, so using them for something else (no matter how well it works conceptually) introduces ambiguities. For example, if you hand a generator to Python’s
for loop, it’s not going to treat it as a coroutine but instead as an iterable.
There were ways to work around this, of course, but it required some odd idioms:
for future in (yield from returns_an_iterable():
value = yield from future
yield from do_something(value)
That works, but it’s a bit strange. Worse, that’s a best case. The idiom gets even stranger when it’s not known ahead of time how many objects will need to be iterated over.
It’s much clearer now in Python 3.5:
async for value in async_iterable:
There’s another problem, related to Python’s special protocol methods such as
__exit__ which are called by the code in the Python interpreter, leaving the programmer with no opportunity to
yield from it. That meant that generator-based coroutines were not compatible with various important bits of Python syntax such as the
with statement. A coroutine couldn’t be called from anything that was called by a special method, whether directly or indirectly, nor was it possible to wait on a future value in a function that was.
The new changes to Python are meant to address these problems. So, what exactly are these changes?
async def is used to define a coroutine. Apart from the
asynckeyword, the syntax is almost identical to a normal
The big differences are, first, that coroutines can contain
async for, and
async with syntaxes, and, second, they are not generators, so they’re not allowed to contain
yield from expressions. It’s impossible for a single function to be both a generator and a coroutine.
await is used to pause the current coroutine until the requested coroutine returns. In other words, an
await expression is just like a function call, except that the called function can pause, allowing the scheduler to run some other thread for a while. If you try to use an
awaitexpression outside of an
async def, Python will raise a syntax error.
async with is used to interface with an asynchronous context manager, which is just like a normal context manager except that instead of
__exit__ methods, it has
__aexit__ coroutine methods. Because they’re coroutines, these methods can do things like wait for data to come in over the network, without blocking the whole program.
async for is used to get data from an asynchronous iterable.
Asynchronous iterables have an
__aiter__ coroutine method which functions like the normal
__iter__ method, but can participate in coroutine scheduling and asynchronous I/O.
__aiter__ should return an object with an
__anext__ coroutine method, which can participate in coroutine scheduling and asynchronous I/O before returning the next iterated value or raising
All of these new features and the convenience they represent are 100% compatible with the existing asyncio scheduler.
Further, as long as you used the @asyncio.coroutine decorator, your existing
asyncio code is also forward compatible with these features without any overhead.