Asynchronous programming has been gaining a lot of traction in the past few years, and for good reason. Although it can be more difficult than the traditional linear style, it is also much more efficient.
For example, instead of waiting for an HTTP request to finish before continuing execution, with Python async coroutines you can submit the request and do other work that's waiting in a queue while waiting for the HTTP request to finish. It might take a bit more thinking to get the logic right, but you'll be able to handle a lot more work with less resources.
Asynchronicity seems to be a big reason why Node.js so popular for server-side programming. Much of the code we write, especially in heavy IO applications like websites, depends on external resources. This could be anything from a remote database call to POSTing to a REST service. As soon as you ask for any of these resources, your code is waiting around with nothing to do.
With asynchronous programming, you allow your code to handle other tasks while waiting for these other resources to respond.
An asynchronous function in Python is typically called a 'coroutine', which is just a function that uses the
async keyword, or one that is decorated with
@asyncio.coroutine. Either of the functions below would work as a coroutine and are effectively equivalent in type:
import asyncio async def ping_server(ip): pass def load_file(path): pass
In case you ever need to determine if a function is a coroutine or not,
asyncio provides the method
asyncio.iscoroutinefunction(func) that does exactly this for you. Or, if you need to determine if an object returned from a function is a coroutine object, you can use
There are a few ways to actually call a coroutine, one of which is the
yield from method. This was introduced in Python 3.3, and has been improved further in Python 3.5 in the form of
async/await (which we'll get to later).
yield from expression can be used as follows:
import asyncio def get_json(client, url): file_content = yield from load_file('/Users/scott/data.txt')
As you can see,
yield from is being used within a function decorated with
@asyncio.coroutine. If you were to try and use
yield from outside this function, then you'd get error from Python like this:
File "main.py", line 1 file_content = yield from load_file('/Users/scott/data.txt') ^ SyntaxError: 'yield' outside function
In order to use this syntax, it must be within another function (typically with the coroutine decorator).
The newer and cleaner syntax is to use the
async/await keywords. Introduced in Python 3.5,
async is used to declare a function as a coroutine, much like what the
@asyncio.coroutine decorator does. It can be applied to the function by putting it at the front of the definition:
async def ping_server(ip): # ping code here...
To actually call this function, we use
await, instead of
yield from, but in much the same way:
async def ping_local(): return await ping_server('192.168.1.1')
Again, just like
yield from, you can't use this outside of another coroutine, otherwise you'll get a syntax error.
In Python 3.5, both ways of calling coroutines are supported, but the
async/await way is meant to be the primary syntax.
Running the event loop
None of the coroutine stuff I described above will matter (or work) if you don't know how to start and run an event loop. The event loop is the central point of execution for asynchronous functions, so when you want to actually execute the coroutine, this is what you'll use.
The event loop provides quite a few features to you:
- Register, execute, and cancel delayed calls (asynchronous functions)
- Create client and server transports for communication
- Create subprocesses and transports for communication with another program
- Delegate function calls to a pool of threads
While there are actually quite a few configurations and event loop types you can use, most of the programs you write will just need to use something like this to schedule a function:
import asyncio async def speak_async(): print('OMG asynchronicity!') loop = asyncio.get_event_loop() loop.run_until_complete(speak_async()) loop.close()
The last three lines are what we're interested in here. It starts by getting the default event loop (
asyncio.get_event_loop()), scheduling and running the async task, and then closing the loop when the loop is done running.
loop.run_until_complete() function is actually blocking, so it won't return until all of the asynchronous methods are done. Since we're only running this on a single thread, there is no way it can move forward while the loop is in progress.
Now, you might think this isn't very useful since we end up blocking on the event loop anyway (instead of just the IO calls), but imagine wrapping your entire program in an async function, which would then allow you to run many asynchronous requests at the same time, like on a web server.
You could even break off the event loop in to its own thread, letting it handle all of the long IO requests while the main thread handles the program logic or UI.
Okay, so let's see a slightly bigger example that we can actually run. The following code is a pretty simple asynchronous program that fetches JSON from Reddit, parses the JSON, and prints out the top posts of the day from /r/python, /r/programming, and /r/compsci.
The first method shown,
get_json(), is called by
get_reddit_top() and just creates an HTTP GET request to the appropriate Reddit URL. When this is called with
await, the event loop can then continue on and service other coroutines while waiting for the HTTP response to get back. Once it does, the JSON is returned to
get_reddit_top(), gets parsed, and is printed out.
import signal import sys import asyncio import aiohttp import json loop = asyncio.get_event_loop() client = aiohttp.ClientSession(loop=loop) async def get_json(client, url): async with client.get(url) as response: assert response.status == 200 return await response.read() async def get_reddit_top(subreddit, client): data1 = await get_json(client, 'https://www.reddit.com/r/' + subreddit + '/top.json?sort=top&t=day&limit=5') j = json.loads(data1.decode('utf-8')) for i in j['data']['children']: score = i['data']['score'] title = i['data']['title'] link = i['data']['url'] print(str(score) + ': ' + title + ' (' + link + ')') print('DONE:', subreddit + '\n') def signal_handler(signal, frame): loop.stop() client.close() sys.exit(0) signal.signal(signal.SIGINT, signal_handler) asyncio.ensure_future(get_reddit_top('python', client)) asyncio.ensure_future(get_reddit_top('programming', client)) asyncio.ensure_future(get_reddit_top('compsci', client)) loop.run_forever()
This is a bit different than the sample code we showed earlier. In order to get multiple coroutines running on the event loop, we're using
asyncio.ensure_future() and then run the loop forever to process everything.
To run this, you'll need to install
aiohttp first, which you can do with PIP:
$ pip install aiohttp
Now just make sure you run it with Python 3.5 or higher, and you should get an output like this:
$ python main.py 46: Python async/await Tutorial (http://stackabuse.com/python-async-await-tutorial/) 16: Using game theory (and Python) to explain the dilemma of exchanging gifts. Turns out: giving a gift probably feels better than receiving one... (http://vknight.org/unpeudemath/code/2015/12/15/The-Prisoners-Dilemma-of-Christmas-Gifts/) 56: Which version of Python do you use? (This is a poll to compare the popularity of Python 2 vs. Python 3) (http://strawpoll.me/6299023) DONE: python 71: The Semantics of Version Control - Wouter Swierstra (http://www.staff.science.uu.nl/~swier004/Talks/vc-semantics-15.pdf) 25: Favorite non-textbook CS books (https://www.reddit.com/r/compsci/comments/3xag9e/favorite_nontextbook_cs_books/) 13: CompSci Weekend SuperThread (December 18, 2015) (https://www.reddit.com/r/compsci/comments/3xacch/compsci_weekend_superthread_december_18_2015/) DONE: compsci 1752: 684.8 TB of data is up for grabs due to publicly exposed MongoDB databases (https://blog.shodan.io/its-still-the-data-stupid/) 773: Instagram's Million Dollar Bug? (http://exfiltrated.com/research-Instagram-RCE.php) 387: Amazingly simple explanation of Diffie-Hellman. His channel has tons of amazing videos and only a few views :( thought I would share! (https://www.youtube.com/watch?v=Afyqwc96M1Y) DONE: programming
Notice that if you run this a few times, the order in which the subreddit data is printed out changes. This is because each of the calls we make releases (yields) control of the thread, allowing another HTTP call to process. Whichever one returns first gets printed out first.
What do you think of Python's async/await? How have you used it in the past? Let us know in the comments!