Asynchronous web-scraping

This asyncio code requires an installation of a few Python asynchronous network libraries, such as aiohttp and aiodns. They are mentioned in the docstring.

Here's the code for the asynchronous scraper at src/extras/async.py; it is structured to be as close as possible to the synchronous version so that it's easier to compare:

"""Asynchronously download a list of webpages and time it Dependencies: Make sure you install aiohttp pip install aiohttp aiodns """ import asyncio import aiohttp from time import time sites = [ "http://news.ycombinator.com/", "https://www.yahoo.com/", "http://www.aliexpress.com/", "http://deelay.me/5000/http://deelay.me/", ] async def find_size(session, url): async with session.get(url) as ...

Get Django Design Patterns and Best Practices - Second Edition now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.