标准的同步爬虫脚本会将大部分运行时间浪费在等待服务器响应上。大规模爬虫时，这会导致它们速度极慢。通过使用异步使用aiohttp并发处理请求，您的脚本会继续处理下一个 URL 而不是停止，从而显著提高任务速度。

在本网络爬虫教程中，我们将逐步讲解如何设置环境、构建异步爬虫、比较性能基准以及向代码添加生产级功能。

异步网页抓取 - 快速概述

传统的（同步）网页抓取方式会迫使脚本在每个页面加载期间处于空闲状态。而异步编程则允许单个 CPU 线程同时处理多个请求，而不是逐个处理。

使用async def和await，你可以明确地告诉 Python 连接暂停发生在哪里。这使得中央事件循环能够在正在进行的下载之间无缝切换。

异步网络爬虫改变了你处理 HTTP 请求的方式。由于不再阻塞网络 I/O，脚本运行速度通常会提升数倍。

在我们的技术栈中，我们将使用aiohttp并发获取页面，并使用 BeautifulSoup 解析 HTML，这将为大规模拉取数据提供实用的基础。

项目设置

为了避免后续出现依赖冲突，您需要先打好基础。在安装工具之前，请务必将项目隔离在虚拟环境中。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#9cdcfe"><code class="language-javascript"># macOS<span style="color:#d4d4d4">/</span>Linux
mkdir async<span style="color:#d4d4d4">-</span>scraper <span style="color:#d4d4d4">&&</span> cd async<span style="color:#d4d4d4">-</span>scraper
python3 <span style="color:#d4d4d4">-</span>m venv venv
source venv<span style="color:#d4d4d4">/</span>bin<span style="color:#d4d4d4">/</span>activate

# Windows
mkdir async<span style="color:#d4d4d4">-</span>scraper <span style="color:#d4d4d4">&&</span> cd async<span style="color:#d4d4d4">-</span>scraper
python <span style="color:#d4d4d4">-</span>m venv venv
venv\Scripts\activate</code>
</span></span></span></span>

现在，让我们安装一些核心库，它们将承担繁重的工作。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#9cdcfe"><code class="language-javascript">pip install aiohttp beautifulsoup4 lxml python<span style="color:#d4d4d4">-</span>dotenv</code>
</span></span></span></span>

Asyncio 和 Aiohttp 基础知识

在开始编写代码之前，我们先来看看异步 HTTP 请求的工作原理：

  • async def。将函数转换为协程，协程可以暂停其执行以让其他代码运行。
  • await。它精确地指示脚本应该在何处将控制权交还给事件循环（通常放在网络调用之前）。
  • asyncio.create_task().调度一个协程在后台运行，同时你的主脚本设置其他任务。
  • asyncio.run(main())启动事件循环并执行你刚刚安排的所有操作。

虽然asyncio负责编排，但实际建立连接的工作则由aiohttp及其强大的ClientSession对象完成。从头开始建立新连接会带来巨大的性能损失。

为了避免这种情况，ClientSession使用Keep-Alive机制保持一个底层 TCP 连接池的开放状态，从而允许您对数千个连续请求重复使用相同的套接字。

一步一步构建异步爬虫

我们从 Books to Scrape 获取数据，因为它为使用 Python 进行网络爬虫提供了一个可靠的沙箱环境，而不会给企业带来巨大压力。

从 CSV 文件加载 URL

将链接直接硬编码到 Python 脚本中，规模扩大后很快就会变成维护噩梦。相反，使用简单的urls.csv文件管理目标链接，就可以使用 Python 内置的csv模块轻松加载它们。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#d4d4d4"><code class="language-python"><span style="color:#569cd6">import</span> csv

<span style="color:#569cd6">def</span> <span style="color:#dcdcaa">create_sample_csv</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    urls <span style="color:#d4d4d4">=</span> <span style="color:#d4d4d4">[</span><span style="color:#ce9178">f"https://books.toscrape.com/catalogue/page-</span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>i<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">.html"</span> <span style="color:#569cd6">for</span> i <span style="color:#569cd6">in</span> <span style="color:#ce9178">range</span><span style="color:#d4d4d4">(</span><span style="color:#b5cea8">1</span><span style="color:#d4d4d4">,</span> <span style="color:#b5cea8">51</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">]</span>
    <span style="color:#569cd6">with</span> <span style="color:#ce9178">open</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">,</span> <span style="color:#ce9178">"w"</span><span style="color:#d4d4d4">,</span> newline<span style="color:#d4d4d4">=</span><span style="color:#ce9178">""</span><span style="color:#d4d4d4">,</span> encoding<span style="color:#d4d4d4">=</span><span style="color:#ce9178">"utf-8"</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> f<span style="color:#d4d4d4">:</span>
        writer <span style="color:#d4d4d4">=</span> csv<span style="color:#d4d4d4">.</span>writer<span style="color:#d4d4d4">(</span>f<span style="color:#d4d4d4">)</span>
        writer<span style="color:#d4d4d4">.</span>writerow<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">[</span><span style="color:#ce9178">"url"</span><span style="color:#d4d4d4">]</span><span style="color:#d4d4d4">)</span>              <span style="color:#6a9955"># header</span>
        writer<span style="color:#d4d4d4">.</span>writerows<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">[</span>u<span style="color:#d4d4d4">]</span> <span style="color:#569cd6">for</span> u <span style="color:#569cd6">in</span> urls<span style="color:#d4d4d4">)</span>

<span style="color:#569cd6">def</span> <span style="color:#dcdcaa">load_urls</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    <span style="color:#569cd6">with</span> <span style="color:#ce9178">open</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">,</span> newline<span style="color:#d4d4d4">=</span><span style="color:#ce9178">""</span><span style="color:#d4d4d4">,</span> encoding<span style="color:#d4d4d4">=</span><span style="color:#ce9178">"utf-8"</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> f<span style="color:#d4d4d4">:</span>
        reader <span style="color:#d4d4d4">=</span> csv<span style="color:#d4d4d4">.</span>DictReader<span style="color:#d4d4d4">(</span>f<span style="color:#d4d4d4">)</span>            <span style="color:#6a9955"># reads the 'url' column by name</span>
        <span style="color:#569cd6">return</span> <span style="color:#d4d4d4">[</span>row<span style="color:#d4d4d4">[</span><span style="color:#ce9178">"url"</span><span style="color:#d4d4d4">]</span> <span style="color:#569cd6">for</span> row <span style="color:#569cd6">in</span> reader <span style="color:#569cd6">if</span> row<span style="color:#d4d4d4">.</span>get<span style="color:#d4d4d4">(</span><span style="color:#ce9178">"url"</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">]</span>

create_sample_csv<span style="color:#d4d4d4">(</span><span style="color:#ce9178">"urls.csv"</span><span style="color:#d4d4d4">)</span>
urls <span style="color:#d4d4d4">=</span> load_urls<span style="color:#d4d4d4">(</span><span style="color:#ce9178">"urls.csv"</span><span style="color:#d4d4d4">)</span>
<span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span>urls<span style="color:#d4d4d4">)</span></code>
</span></span></span></span>

主函数和事件循环

将核心数据获取逻辑封装在异步的 `def main()`函数中，可以让你拥有一个清晰的入口点来管理当前会话。为了在快速获取数据的同时还能与目标服务器良好兼容，你应该使用信号量来限制并发。

然后，asyncio.gather()运行所有获取任务并等待结果。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#d4d4d4"><code class="language-python"><span style="color:#569cd6">import</span> asyncio
<span style="color:#569cd6">import</span> aiohttp
<span style="color:#569cd6">import</span> csv

<span style="color:#569cd6">def</span> <span style="color:#dcdcaa">load_urls</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    <span style="color:#569cd6">with</span> <span style="color:#ce9178">open</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">,</span> newline<span style="color:#d4d4d4">=</span><span style="color:#ce9178">""</span><span style="color:#d4d4d4">,</span> encoding<span style="color:#d4d4d4">=</span><span style="color:#ce9178">"utf-8"</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> f<span style="color:#d4d4d4">:</span>
        reader <span style="color:#d4d4d4">=</span> csv<span style="color:#d4d4d4">.</span>DictReader<span style="color:#d4d4d4">(</span>f<span style="color:#d4d4d4">)</span>            <span style="color:#6a9955"># reads the 'url' column by name</span>
        <span style="color:#569cd6">return</span> <span style="color:#d4d4d4">[</span>row<span style="color:#d4d4d4">[</span><span style="color:#ce9178">"url"</span><span style="color:#d4d4d4">]</span> <span style="color:#569cd6">for</span> row <span style="color:#569cd6">in</span> reader <span style="color:#569cd6">if</span> row<span style="color:#d4d4d4">.</span>get<span style="color:#d4d4d4">(</span><span style="color:#ce9178">"url"</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">]</span>


<span style="color:#569cd6">async</span> <span style="color:#569cd6">def</span> <span style="color:#dcdcaa">fetch</span><span style="color:#d4d4d4">(</span>session<span style="color:#d4d4d4">,</span> url<span style="color:#d4d4d4">,</span> semaphore<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> semaphore<span style="color:#d4d4d4">:</span>                      <span style="color:#6a9955"># cap concurrency at 20</span>
        <span style="color:#569cd6">try</span><span style="color:#d4d4d4">:</span>
            <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> session<span style="color:#d4d4d4">.</span>get<span style="color:#d4d4d4">(</span>url<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> response<span style="color:#d4d4d4">:</span>
                response<span style="color:#d4d4d4">.</span>raise_for_status<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>    <span style="color:#6a9955"># turn 4xx/5xx into an exception</span>
                <span style="color:#569cd6">return</span> <span style="color:#569cd6">await</span> response<span style="color:#d4d4d4">.</span>text<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>
        <span style="color:#569cd6">except</span> aiohttp<span style="color:#d4d4d4">.</span>ClientError <span style="color:#569cd6">as</span> e<span style="color:#d4d4d4">:</span>
            <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Failed to fetch </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>url<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">: </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>e<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">"</span><span style="color:#d4d4d4">)</span>
            <span style="color:#569cd6">return</span> <span style="color:#569cd6">None</span>
        <span style="color:#569cd6">except</span> <span style="color:#d4d4d4">(</span>aiohttp<span style="color:#d4d4d4">.</span>ClientError<span style="color:#d4d4d4">,</span> asyncio<span style="color:#d4d4d4">.</span>TimeoutError<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> e<span style="color:#d4d4d4">:</span>
            <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Failed to fetch </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>url<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">: </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>e<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">"</span><span style="color:#d4d4d4">)</span>
            <span style="color:#569cd6">return</span> <span style="color:#569cd6">None</span>


<span style="color:#569cd6">async</span> <span style="color:#569cd6">def</span> <span style="color:#dcdcaa">main</span><span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    urls <span style="color:#d4d4d4">=</span> load_urls<span style="color:#d4d4d4">(</span><span style="color:#ce9178">'urls.csv'</span><span style="color:#d4d4d4">)</span>
    semaphore <span style="color:#d4d4d4">=</span> asyncio<span style="color:#d4d4d4">.</span>Semaphore<span style="color:#d4d4d4">(</span><span style="color:#b5cea8">20</span><span style="color:#d4d4d4">)</span>
    timeout <span style="color:#d4d4d4">=</span> aiohttp<span style="color:#d4d4d4">.</span>ClientTimeout<span style="color:#d4d4d4">(</span>total<span style="color:#d4d4d4">=</span><span style="color:#b5cea8">30</span><span style="color:#d4d4d4">)</span>
    <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> aiohttp<span style="color:#d4d4d4">.</span>ClientSession<span style="color:#d4d4d4">(</span>timeout<span style="color:#d4d4d4">=</span>timeout<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> session<span style="color:#d4d4d4">:</span>
        tasks <span style="color:#d4d4d4">=</span> <span style="color:#d4d4d4">[</span>fetch<span style="color:#d4d4d4">(</span>session<span style="color:#d4d4d4">,</span> url<span style="color:#d4d4d4">,</span> semaphore<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">for</span> url <span style="color:#569cd6">in</span> urls<span style="color:#d4d4d4">]</span>
        results <span style="color:#d4d4d4">=</span> <span style="color:#569cd6">await</span> asyncio<span style="color:#d4d4d4">.</span>gather<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">*</span>tasks<span style="color:#d4d4d4">)</span>
        <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Finished fetching </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span><span style="color:#ce9178">len</span><span style="color:#d4d4d4">(</span>results<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">}</span></span><span style="color:#ce9178"> pages."</span><span style="color:#d4d4d4">)</span>


<span style="color:#569cd6">if</span> __name__ <span style="color:#d4d4d4">==</span> <span style="color:#ce9178">'__main__'</span><span style="color:#d4d4d4">:</span>
    asyncio<span style="color:#d4d4d4">.</span>run<span style="color:#d4d4d4">(</span>main<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">)</span></code>
</span></span></span></span>

获取函数

大规模应用时，网络问题不可避免。这是对上述核心获取函数的升级，增加了一个使用指数退避和超时机制的重试块，以处理速率限制和瞬态错误。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#d4d4d4"><code class="language-python"><span style="color:#569cd6">async</span> <span style="color:#569cd6">def</span> <span style="color:#dcdcaa">fetch</span><span style="color:#d4d4d4">(</span>session<span style="color:#d4d4d4">,</span> url<span style="color:#d4d4d4">,</span> semaphore<span style="color:#d4d4d4">,</span> max_retries<span style="color:#d4d4d4">=</span><span style="color:#b5cea8">3</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> semaphore<span style="color:#d4d4d4">:</span>
        <span style="color:#569cd6">for</span> attempt <span style="color:#569cd6">in</span> <span style="color:#ce9178">range</span><span style="color:#d4d4d4">(</span>max_retries<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
            <span style="color:#569cd6">try</span><span style="color:#d4d4d4">:</span>
                <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> session<span style="color:#d4d4d4">.</span>get<span style="color:#d4d4d4">(</span>url<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> response<span style="color:#d4d4d4">:</span>
                    <span style="color:#6a9955"># Retry on transient server errors and rate limiting</span>
                    <span style="color:#569cd6">if</span> response<span style="color:#d4d4d4">.</span>status <span style="color:#569cd6">in</span> <span style="color:#d4d4d4">(</span><span style="color:#b5cea8">429</span><span style="color:#d4d4d4">,</span> <span style="color:#b5cea8">502</span><span style="color:#d4d4d4">,</span> <span style="color:#b5cea8">503</span><span style="color:#d4d4d4">,</span> <span style="color:#b5cea8">504</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
                        <span style="color:#569cd6">if</span> attempt <span style="color:#d4d4d4"><</span> max_retries <span style="color:#d4d4d4">-</span> <span style="color:#b5cea8">1</span><span style="color:#d4d4d4">:</span>
                            <span style="color:#569cd6">await</span> asyncio<span style="color:#d4d4d4">.</span>sleep<span style="color:#d4d4d4">(</span><span style="color:#b5cea8">2</span> <span style="color:#d4d4d4">**</span> attempt<span style="color:#d4d4d4">)</span>
                            <span style="color:#569cd6">continue</span>
                        <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Gave up on </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>url<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">: status </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>response<span style="color:#d4d4d4">.</span>status<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">"</span><span style="color:#d4d4d4">)</span>
                        <span style="color:#569cd6">return</span> <span style="color:#569cd6">None</span>
                    response<span style="color:#d4d4d4">.</span>raise_for_status<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>
                    <span style="color:#569cd6">return</span> <span style="color:#569cd6">await</span> response<span style="color:#d4d4d4">.</span>text<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>
            <span style="color:#569cd6">except</span> <span style="color:#d4d4d4">(</span>aiohttp<span style="color:#d4d4d4">.</span>ClientError<span style="color:#d4d4d4">,</span> asyncio<span style="color:#d4d4d4">.</span>TimeoutError<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> e<span style="color:#d4d4d4">:</span>
                <span style="color:#569cd6">if</span> attempt <span style="color:#d4d4d4"><</span> max_retries <span style="color:#d4d4d4">-</span> <span style="color:#b5cea8">1</span><span style="color:#d4d4d4">:</span>
                    <span style="color:#569cd6">await</span> asyncio<span style="color:#d4d4d4">.</span>sleep<span style="color:#d4d4d4">(</span><span style="color:#b5cea8">2</span> <span style="color:#d4d4d4">**</span> attempt<span style="color:#d4d4d4">)</span>
                    <span style="color:#569cd6">continue</span>
                <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Failed </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>url<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">: </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>e<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">"</span><span style="color:#d4d4d4">)</span>
                <span style="color:#569cd6">return</span> <span style="color:#569cd6">None</span>
    <span style="color:#569cd6">return</span> <span style="color:#569cd6">None</span>  <span style="color:#6a9955"># unreachable in practice, but makes the contract explicit</span></code>
</span></span></span></span>

使用 BeautifulSoup 解析 HTML

获取到原始 HTML 代码后，就可以使用标准的网页抓取技术了。这里需要用到 BeautifulSoup 和 CSS 选择器来提取标题、价格和库存信息。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#d4d4d4"><code class="language-python"><span style="color:#569cd6">from</span> bs4 <span style="color:#569cd6">import</span> BeautifulSoup

<span style="color:#569cd6">def</span> <span style="color:#dcdcaa">parse_book</span><span style="color:#d4d4d4">(</span>html<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    <span style="color:#569cd6">if</span> <span style="color:#569cd6">not</span> html<span style="color:#d4d4d4">:</span>
        <span style="color:#569cd6">return</span> <span style="color:#569cd6">None</span>
    soup <span style="color:#d4d4d4">=</span> BeautifulSoup<span style="color:#d4d4d4">(</span>html<span style="color:#d4d4d4">,</span> <span style="color:#ce9178">'lxml'</span><span style="color:#d4d4d4">)</span>

    title_el <span style="color:#d4d4d4">=</span> soup<span style="color:#d4d4d4">.</span>select_one<span style="color:#d4d4d4">(</span><span style="color:#ce9178">'h1'</span><span style="color:#d4d4d4">)</span>
    price_el <span style="color:#d4d4d4">=</span> soup<span style="color:#d4d4d4">.</span>select_one<span style="color:#d4d4d4">(</span><span style="color:#ce9178">'.price_color'</span><span style="color:#d4d4d4">)</span>
    stock_el <span style="color:#d4d4d4">=</span> soup<span style="color:#d4d4d4">.</span>select_one<span style="color:#d4d4d4">(</span><span style="color:#ce9178">'.instock.availability'</span><span style="color:#d4d4d4">)</span>

    <span style="color:#569cd6">return</span> <span style="color:#d4d4d4">{</span>
        <span style="color:#ce9178">'title'</span><span style="color:#d4d4d4">:</span> title_el<span style="color:#d4d4d4">.</span>text <span style="color:#569cd6">if</span> title_el <span style="color:#569cd6">else</span> <span style="color:#ce9178">'Unknown'</span><span style="color:#d4d4d4">,</span>
        <span style="color:#ce9178">'price'</span><span style="color:#d4d4d4">:</span> price_el<span style="color:#d4d4d4">.</span>text<span style="color:#d4d4d4">.</span>strip<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">if</span> price_el <span style="color:#569cd6">else</span> <span style="color:#ce9178">'0.00'</span><span style="color:#d4d4d4">,</span>
        <span style="color:#ce9178">'stock'</span><span style="color:#d4d4d4">:</span> stock_el<span style="color:#d4d4d4">.</span>text<span style="color:#d4d4d4">.</span>strip<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">if</span> stock_el <span style="color:#569cd6">else</span> <span style="color:#ce9178">'Unknown'</span><span style="color:#d4d4d4">,</span>
    <span style="color:#d4d4d4">}</span></code>
</span></span></span></span>

将结果保存为 JSON

将数据转储到标准 JSON 文件中适用于小规模操作，但对于大规模操作，建议使用 JSON Lines (JSONL)，以便可以按流追加记录。如果处理的是海量数据集，建议为每个产品写入单独的文件，以避免脚本崩溃时数据丢失。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#d4d4d4"><code class="language-python"><span style="color:#569cd6">import</span> json

<span style="color:#569cd6">def</span> <span style="color:#dcdcaa">save_json</span><span style="color:#d4d4d4">(</span>records<span style="color:#d4d4d4">,</span> filepath<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    records <span style="color:#d4d4d4">=</span> <span style="color:#d4d4d4">[</span>r <span style="color:#569cd6">for</span> r <span style="color:#569cd6">in</span> records <span style="color:#569cd6">if</span> r<span style="color:#d4d4d4">]</span>
    <span style="color:#569cd6">with</span> <span style="color:#ce9178">open</span><span style="color:#d4d4d4">(</span>filepath<span style="color:#d4d4d4">,</span> <span style="color:#ce9178">"w"</span><span style="color:#d4d4d4">,</span> encoding<span style="color:#d4d4d4">=</span><span style="color:#ce9178">"utf-8"</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> f<span style="color:#d4d4d4">:</span>
        json<span style="color:#d4d4d4">.</span>dump<span style="color:#d4d4d4">(</span>records<span style="color:#d4d4d4">,</span> f<span style="color:#d4d4d4">,</span> indent<span style="color:#d4d4d4">=</span><span style="color:#b5cea8">2</span><span style="color:#d4d4d4">,</span> ensure_ascii<span style="color:#d4d4d4">=</span><span style="color:#569cd6">False</span><span style="color:#d4d4d4">)</span>
    <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Saved </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span><span style="color:#ce9178">len</span><span style="color:#d4d4d4">(</span>records<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">}</span></span><span style="color:#ce9178"> records to </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>filepath<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">"</span><span style="color:#d4d4d4">)</span></code>
</span></span></span></span>

基准测试 - 异步与同步

将原始数据与自己的代码进行对比，可以证明异步网络爬虫的巨大优势，尤其是在与传统脚本的线性爬取进行直接比较时。

标准的同步方法会逐个遍历列表。即使通过会话重用连接，脚本仍然会完全阻塞，等待服务器生成并发送 HTML 有效负载后才能处理下一个项目。

<span style="color:#1a2d2f"><span style="background-color:#f8f8f8"><span style="background-color:#004047"><span style="color:#d4d4d4"><code class="language-python"><span style="color:#569cd6">import</span> asyncio
<span style="color:#569cd6">import</span> time
<span style="color:#569cd6">import</span> requests
<span style="color:#569cd6">import</span> aiohttp


<span style="color:#569cd6">def</span> <span style="color:#dcdcaa">scraper_sync</span><span style="color:#d4d4d4">(</span>urls<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    results <span style="color:#d4d4d4">=</span> <span style="color:#d4d4d4">[</span><span style="color:#d4d4d4">]</span>
    <span style="color:#569cd6">with</span> requests<span style="color:#d4d4d4">.</span>Session<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> session<span style="color:#d4d4d4">:</span>
        <span style="color:#569cd6">for</span> url <span style="color:#569cd6">in</span> urls<span style="color:#d4d4d4">:</span>
            response <span style="color:#d4d4d4">=</span> session<span style="color:#d4d4d4">.</span>get<span style="color:#d4d4d4">(</span>url<span style="color:#d4d4d4">,</span> timeout<span style="color:#d4d4d4">=</span><span style="color:#b5cea8">30</span><span style="color:#d4d4d4">)</span>
            results<span style="color:#d4d4d4">.</span>append<span style="color:#d4d4d4">(</span>response<span style="color:#d4d4d4">.</span>text<span style="color:#d4d4d4">)</span>
    <span style="color:#569cd6">return</span> results


<span style="color:#569cd6">async</span> <span style="color:#569cd6">def</span> <span style="color:#dcdcaa">scraper_async</span><span style="color:#d4d4d4">(</span>urls<span style="color:#d4d4d4">,</span> concurrency<span style="color:#d4d4d4">=</span><span style="color:#b5cea8">20</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
    semaphore <span style="color:#d4d4d4">=</span> asyncio<span style="color:#d4d4d4">.</span>Semaphore<span style="color:#d4d4d4">(</span>concurrency<span style="color:#d4d4d4">)</span>
    timeout <span style="color:#d4d4d4">=</span> aiohttp<span style="color:#d4d4d4">.</span>ClientTimeout<span style="color:#d4d4d4">(</span>total<span style="color:#d4d4d4">=</span><span style="color:#b5cea8">30</span><span style="color:#d4d4d4">)</span>

    <span style="color:#569cd6">async</span> <span style="color:#569cd6">def</span> <span style="color:#dcdcaa">_fetch</span><span style="color:#d4d4d4">(</span>session<span style="color:#d4d4d4">,</span> url<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">:</span>
        <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> semaphore<span style="color:#d4d4d4">:</span>
            <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> session<span style="color:#d4d4d4">.</span>get<span style="color:#d4d4d4">(</span>url<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> response<span style="color:#d4d4d4">:</span>
                <span style="color:#569cd6">return</span> <span style="color:#569cd6">await</span> response<span style="color:#d4d4d4">.</span>text<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>

    <span style="color:#569cd6">async</span> <span style="color:#569cd6">with</span> aiohttp<span style="color:#d4d4d4">.</span>ClientSession<span style="color:#d4d4d4">(</span>timeout<span style="color:#d4d4d4">=</span>timeout<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">as</span> session<span style="color:#d4d4d4">:</span>
        <span style="color:#569cd6">return</span> <span style="color:#569cd6">await</span> asyncio<span style="color:#d4d4d4">.</span>gather<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">*</span><span style="color:#d4d4d4">[</span>_fetch<span style="color:#d4d4d4">(</span>session<span style="color:#d4d4d4">,</span> url<span style="color:#d4d4d4">)</span> <span style="color:#569cd6">for</span> url <span style="color:#569cd6">in</span> urls<span style="color:#d4d4d4">]</span><span style="color:#d4d4d4">)</span>


<span style="color:#569cd6">if</span> __name__ <span style="color:#d4d4d4">==</span> <span style="color:#ce9178">"__main__"</span><span style="color:#d4d4d4">:</span>
    urls <span style="color:#d4d4d4">=</span> <span style="color:#d4d4d4">[</span><span style="color:#ce9178">f"https://books.toscrape.com/catalogue/page-</span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>i<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">.html"</span>
            <span style="color:#569cd6">for</span> i <span style="color:#569cd6">in</span> <span style="color:#ce9178">range</span><span style="color:#d4d4d4">(</span><span style="color:#b5cea8">1</span><span style="color:#d4d4d4">,</span> <span style="color:#b5cea8">51</span><span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">]</span>

    start <span style="color:#d4d4d4">=</span> time<span style="color:#d4d4d4">.</span>perf_counter<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>
    scraper_sync<span style="color:#d4d4d4">(</span>urls<span style="color:#d4d4d4">)</span>
    sync_time <span style="color:#d4d4d4">=</span> time<span style="color:#d4d4d4">.</span>perf_counter<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span> <span style="color:#d4d4d4">-</span> start

    start <span style="color:#d4d4d4">=</span> time<span style="color:#d4d4d4">.</span>perf_counter<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span>
    asyncio<span style="color:#d4d4d4">.</span>run<span style="color:#d4d4d4">(</span>scraper_async<span style="color:#d4d4d4">(</span>urls<span style="color:#d4d4d4">)</span><span style="color:#d4d4d4">)</span>
    async_time <span style="color:#d4d4d4">=</span> time<span style="color:#d4d4d4">.</span>perf_counter<span style="color:#d4d4d4">(</span><span style="color:#d4d4d4">)</span> <span style="color:#d4d4d4">-</span> start

    <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Sync:  </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>sync_time<span style="color:#d4d4d4">:</span>.2f<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">s"</span><span style="color:#d4d4d4">)</span>
    <span style="color:#569cd6">print</span><span style="color:#d4d4d4">(</span><span style="color:#ce9178">f"Async: </span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>async_time<span style="color:#d4d4d4">:</span>.2f<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">s (</span><span style="color:#9cdcfe"><span style="color:#d4d4d4">{</span>sync_time <span style="color:#d4d4d4">/</span> async_time<span style="color:#d4d4d4">:</span>.1f<span style="color:#d4d4d4">}</span></span><span style="color:#ce9178">x faster)"</span><span style="color:#d4d4d4">)</span></code>
</span></span></span></span>

使用`time.perf_counter()`来测量执行时间可以明显看出性能差距。同步处理一百个 URL 需要四十秒，而异步代码只需两三秒即可完成。

实际速度取决于网络延迟和服务器响应时间，但如果抓取的 URL 超过几个，同步请求总是会成为管道的瓶颈。

稳健的错误处理和可靠性

如果你想让你的爬虫程序无人值守运行，你需要编写逻辑来预判它会在最糟糕的时刻出现故障。设计一个生产环境的爬虫程序需要你预见到各种各样的实际故障：

  • DNS解析失败，导致目标服务器完全无法定位。
  • 连接重置导致下载过程中断开套接字连接。
  • 当对端点的访问过于频繁时，会采取严格的 429 速率限制。
  • 后端过载触发的随机 5xx 错误。
  • 静默布局更新会突然改变您的 CSS 选择器。

当这些故障发生时，使用 Python 的日志库捕获准确的上下文信息可以避免很多麻烦。配置日志记录器以自动包含时间戳，并显式记录目标 URL、状态码和错误类型。将失败的 URL 附加到单独的文本文件中，还可以确保以后可以轻松地重新运行它们。

通过实施三次重试限制并采用指数退避算法，几乎可以捕获所有瞬态错误，而不会对服务器造成过大压力。但是，如果遇到 429 状态码，重试就不够了，因为暂停工作进程以响应服务器的 Retry-After 标头是避免永久 IP 封禁的唯一可持续方法。

# python # 爬虫 # 开发语言
Logo
CSDN-OPC开发者社区

这里是“一人公司”的成长家园。我们提供从产品曝光、技术变现到法律财税的全栈内容，并连接云服务、办公空间等稀缺资源，助你专注创造，无忧运营。

更多推荐

cover

OPC一人公司轻创业实战解析：摆脱低效内耗，系统化实现个体盈利

avatar CSDN-OPC开发者社区
cover

DeepSeek Harness 深度揭秘：5 个改写 AI 智能体游戏规则的核心真相

avatar CSDN-OPC开发者社区
cover

小龙虾Windows版快速安装上手，TopClaw零代码满血直通飞书与微信

avatar CSDN-OPC开发者社区