惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Security Archives - TechRepublic
Security Archives - TechRepublic
Project Zero
Project Zero
K
Kaspersky official blog
G
Google Developers Blog
T
Threat Research - Cisco Blogs
T
The Blog of Author Tim Ferriss
Cyberwarzone
Cyberwarzone
Y
Y Combinator Blog
Recorded Future
Recorded Future
Blog — PlanetScale
Blog — PlanetScale
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
Cisco Talos Blog
Cisco Talos Blog
Latest news
Latest news
Microsoft Security Blog
Microsoft Security Blog
H
Help Net Security
S
Schneier on Security
P
Palo Alto Networks Blog
H
Hacker News: Front Page
N
News and Events Feed by Topic
N
Netflix TechBlog - Medium
博客园 - Franky
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
SecWiki News
SecWiki News
Cloudbric
Cloudbric
TaoSecurity Blog
TaoSecurity Blog
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
The Hacker News
The Hacker News
C
Check Point Blog
L
LangChain Blog
腾讯CDC
小众软件
小众软件
T
Tenable Blog
Google DeepMind News
Google DeepMind News
GbyAI
GbyAI
L
LINUX DO - 最新话题
A
About on SuperTechFans
Google Online Security Blog
Google Online Security Blog
C
Cisco Blogs
Recent Announcements
Recent Announcements
Hacker News: Ask HN
Hacker News: Ask HN
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
Vercel News
Vercel News
雷峰网
雷峰网
美团技术团队
D
DataBreaches.Net
Martin Fowler
Martin Fowler
Help Net Security
Help Net Security
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
F
Full Disclosure
博客园_首页

Pierce Freeman

A browser for agents | Pierce Freeman The grey market of podcast appearances The way I travel | Pierce Freeman Fixing slow AWS uploads | Pierce Freeman Local tools should still use vaults We solved scratch content first Starting a podcast in 2025 Being late but still being early Automating our home video imports Adding my parents to tailscale A deep dive on agent sandboxes Language servers for AI | Pierce Freeman My simple home podcast studio We need centralized infrastructure | Pierce Freeman Coercing agents to follow conventions using AST validation My unified theory of social selling My personal backup strategy | Pierce Freeman July updates to the homelab How the KV Cache works Reputation is becoming everything | Pierce Freeman Building a (kind of) invisible mac app Updated knowledge in language models Making an ascii animation | Pierce Freeman How speculative decoding works | Pierce Freeman Under the hood of Claude Code Doing things because they're easy, not hard Speeding up sideeffects with JIT in mountaineer Firehot for hot reloading in Python Misadventures in Python hot reloading How text diffusion works | Pierce Freeman The tenacity of modern LLMs The ergonomics of rails | Pierce Freeman How language servers work | Pierce Freeman Just add eggs | Pierce Freeman Unfortunately SEO still matters | Pierce Freeman The futility of human-only web requirements Setting up Input Leap | Pierce Freeman Checking in on Waymo | Pierce Freeman The react revolution | Pierce Freeman Speeding up many small transfers to a unifi nas Quick notes on swift libraries AI engineering is a different animal San Francisco | Pierce Freeman Debugging a mountaineer rendering segfault Local network config on macOS Building our home network | Pierce Freeman Introducing Envelope.dev | Pierce Freeman Legacy code and AI copilots Typehinting from day-zero | Pierce Freeman Generating database migrations with acyclic graphs Lofoten | Pierce Freeman Mountaineer v0.1: Webapps in Python and React Constraining LLM Outputs | Pierce Freeman Passthrough above all | Pierce Freeman Accuracy in kudos | Pierce Freeman How quick we are to adapt The curious case of LM repetition Costa Rica | Pierce Freeman Debugging chrome extensions with system-level logging Speeding up runpod | Pierce Freeman Inline footnotes with html templates Parsing Common Crawl in a day for $60 An era of rich CLI All or nothing with remote work The Next 10 Years | Pierce Freeman Adding wheels to flash-attention | Pierce Freeman LLMs as interdisciplinary agents | Pierce Freeman New Zealand | Pierce Freeman Representations in autoregressive models | Pierce Freeman Let's talk about Siri | Pierce Freeman Minimum viable public infrastructure | Pierce Freeman Reasoning vs. Memorization in LLMs Automatically migrate enums in alembic Greater sequence lengths will set us free On learning to ski | Pierce Freeman Dolomites | Pierce Freeman Using grpc with node and typescript Opportunity years | Pierce Freeman Buzzword peaks and valleys | Pierce Freeman Buenos Aires | Pierce Freeman Network routing interaction on MacOS Independent work: November recap | Pierce Freeman Debugging slow pytorch training performance The provenance of copy and paste Debugging tips for neural network training Patagonia | Pierce Freeman Santiago | Pierce Freeman My 2022 digital travel kit AWS vs GCP - GPU Availability V2 Independent work: October recap | Pierce Freeman Planning Patagonia | Pierce Freeman Relationship modeling | Pierce Freeman The power of status updates A new chapter | Pierce Freeman Give my library a coffee shop AWS vs GCP - GPU Availability V1 Switzerland | Pierce Freeman Headfull browsers beat headless | Pierce Freeman Webcrawling tradeoffs | Pierce Freeman Copenhagen | Pierce Freeman
httpx is the right way to do web requests in Python
2025-07-23 · via Pierce Freeman

As even more webapps become shims on top of API services and those services have latency measured in seconds or minutes1, the ergonomics of network requests become pretty important. The gold standard here is probably fetch() in Javascript: it's simple, powerful, and baked into most browsers.2 It's an industry standard for a reason.

Python lacks a stdlib answer with the same kind of adoption. For production applications I think there are three main things you want out of your framework:

  • Async support to avoid blocking the event loop
  • Easily testable so you're not making real requests in your tests
  • OpenAPI client generation support for any internal service communication

Even though aiohttp still wins on raw throughput, my default choice has become httpx. You can nearly always find it in my pyproject.tomls. Here's why.

async clients

I haven't ever used the sync clients provided by httpx, although they provide these too. The compelling functionality is really on the async side.

Technically httpx doesn't handle networking itself - it delegates to httpcore, which provides the low-level HTTP transport implementation. When you create an AsyncClient, httpx instantiates an AsyncHTTPTransport that wraps httpcore's AsyncConnectionPool. That's a mouthful.

When you await client.get(), it:

  1. Acquires a connection from the pool via _assign_requests_to_connections()
  2. Delegates to asyncio - each connection wraps an asyncio.StreamReader/StreamWriter pair
  3. Shares connections in a connection pool3

If you're stuck in a synchronous codebase, you can certainly use threading to provide some concurrency that's similar to asyncio. But you're system limited to only being able to parallelize so many threads at the same time before you start reaching diminishing returns of CPU switching. With asyncio you're really just issuing all network requests up to the kernel / networking card and waiting to receive the signal of the data coming back over the stream. There you're more limited by file descriptors and DNS resolution.

Connection pooling can give a meaningful performance advantage if you're making a lot of repeated requests to a single domain over TLS (which is the default on most of the web these days). The numbers are illustrative. A fresh HTTPS connection still needs:

  1. DNS lookup (~20-100 ms)
  2. TCP handshake (1 RTT, ~50-200 ms depending on geography)
  3. TLS handshake (1 RTT on TLS 1.3, 2 RTTs on TLS 1.2)
  4. HTTP request/response (1 RTT + server time)

That’s half a second of ceremony per socket if you don’t pool. httpx’s AsyncConnectionPool amortises all but #4 after the first hit.

pytest-httpx intercepts

Most HTTP mocking libraries patch at the unittest.mock.patch() level. This ends up being pretty brittle because patches in python need to occur before the in-memory package is imported, otherwise you can often patch the wrong object that won't affect what's in runtime. If you (or your mocking library) aren't careful, you can inadvertently make real outbound requests when you're running unit tests.

Because of the abstraction layers in httpcore transports, pytest-httpx can do something that's more reliable. The AsyncRequestInterface defines a single method that we can override for a fresh interface:

# From httpcore/_async/interfaces.py
class AsyncRequestInterface:
    async def handle_async_request(self, request: Request) -> Response:
        raise NotImplementedError()

pytest-httpx provides a MockTransport that completely replaces the real transport.

class MockTransport:

    async def _handle_async_request(
        self,
        real_transport: httpx.AsyncHTTPTransport,
        request: httpx.Request,
    ) -> httpx.Response:
        # Store the content in request for future matching
        await request.aread()
        self._requests.append((real_transport, request))

        callback = self._get_callback(real_transport, request)
        if callback:
            response = callback(request)

            if response:
                if inspect.isawaitable(response):
                    response = await response
                return _unread(response)

        self._request_not_matched(real_transport, request)

This lets pytest-httpx operate at the same level as the real HTTP transport. Your code paths are identical; the only difference is that MockTransport.handle_async_request() returns your pre-configured response instead of making a real network call.

The API stays the same on the httpx side while being able to swap this transport in to the global singleton anywhere httpx is used.

OpenAPI Support

There's now pretty solid community support for automatically generating a typehinted httpx client from an OpenAPI spec. This isn't baked into httpx directly but their jinja templates rely on httpx for the main IO layer.

Since most modern libraries either give you OpenAPI support out of the box or bake it into the design assumptions, this creates pretty a seamless workflow from API definition to client code.

1. Define your API spec (user-api.yaml):

Some people write these by hand but I've always preferred marking up my webapp code with the output spec inline. It lets you change your API in oneshot instead of having to go through the OpenAPI spec, regenerating a template, and then filling in the service code.4

openapi: 3.0.0
info:
  title: User Management API
  version: 1.0.0
servers:
  - url: https://api.example.com/v1
paths:
  /users:
    get:
      operationId: list_users
      parameters:
        - name: limit
          in: query
          schema:
            type: integer
            default: 10
      responses:
        '200':
          description: List of users
          content:
            application/json:
              schema:
                type: object
                properties:
                  users:
                    type: array
                    items:
                      $ref: '#/components/schemas/User'
components:
  schemas:
    User:
      type: object
      properties:
        id:
          type: string
          format: uuid
        email:
          type: string
          format: email
        name:
          type: string
        created_at:
          type: string
          format: date-time
        is_active:
          type: boolean
      required:
        - id
        - email
        - name
        - created_at
        - is_active

2. Generate the client:

uvx openapi-python-client generate --path user-api.yaml

3. Use the generated client:

And with that one uvx script, you've got yourself a client.

Distinguishing between sync and asyncio - plus the more detailed extensions sync_detailed and asyncio_detailed - require a slightly verbose syntax. But gives you all the flexibility of generic OpenAPI schemas with structured typehinting. I think it's well worth it.

from user_management_api_client import Client
from user_management_api_client.api.default import list_users, get_user
from user_management_api_client.models import User
from uuid import UUID

async with Client(base_url="https://api.example.com/v1") as client:
    # Same methods, but async
    users_response = await list_users.asyncio(client=client, limit=5)

Within this library we have:

  • Full type safety: The User model is a proper dataclass with UUID, datetime, and email validation
  • Both sync and async: Every endpoint gets .sync() and .asyncio() variants just in case you're still stuck in sync land
  • Rich response parsing: Access both parsed data and raw HTTP responses
  • Automatic serialization: JSON request/response handling with proper Python types

It makes for a pretty smooth development loop: define your API contract on the server side, generate type-safe clients, and get all the benefits of httpx's async support and connection pooling without writing any networking code yourself.

Other asyncio libraries

I'm not an extensive expert on all of the other asyncio networking libraries but I have tried a handful of them. Here's how the major alternatives compare against httpx's three key strengths:

LibraryAsyncTesting storyOpenAPI toolingWhen to pick
aiohttpFirst-classManual mocksNone built-inMax raw performance & you control the stack
urllib3– (sync only)PatchingNeed the absolute lowest-level knobs
grequestsGevent monkey-patchRequests-styleSync-onlyQuick wins on legacy code
uplinkBackend-agnosticFollows chosen clientDeclarative, not auto-genRetrofit-style APIs with pluggable transports

Notable omissions: pycurl (C-speed but ugly API), trio’s clients (different concurrency model).

Just use it™

I'm officially5 nominating httpx for the defacto industry standard for web networking in Python in 2025. It feels like the closet alternative to fetch that we have - decent performance, well designed, and easy to test.

We probably would have said the same thing about requests and the related responses mocking library until the async redevelopment of Python threw the package ecosystem upside down. But in the calm after the storm there are some libraries that are becoming that same scope of standard again. I'm here hoping it's httpx.

And if it is, let's see how long it can keep the title.

  1. LLMs agents, I'm looking at you. ↩

  2. Also recently into server side libraries like Node and Bun. The only thing it doesn't really have support for is good progress bar tracking. ↩

  3. http2 mitigates the need for this somewhat by having built-in multiplexing support. ↩

  4. I recognize the arguments that forcing this process is a better way to ensure that your APIs are reverse compatible for old clients. But imo, these checks are better left for static analysis not for encouraging a more circuitous process during development. ↩

  5. Right here, right now. ↩