惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

量子位
Recent Announcements
Recent Announcements
D
Docker
V
V2EX
阮一峰的网络日志
阮一峰的网络日志
Vercel News
Vercel News
Microsoft Security Blog
Microsoft Security Blog
The GitHub Blog
The GitHub Blog
U
Unit 42
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
月光博客
月光博客
腾讯CDC
B
Blog
博客园_首页
罗磊的独立博客
D
DataBreaches.Net
IT之家
IT之家
酷 壳 – CoolShell
酷 壳 – CoolShell
L
LangChain Blog
aimingoo的专栏
aimingoo的专栏
MongoDB | Blog
MongoDB | Blog
GbyAI
GbyAI
Stack Overflow Blog
Stack Overflow Blog
M
MIT News - Artificial intelligence

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant
From Browser cURL to Clean Python Requests Code, Step by ...
zhenye yu · 2026-06-19 · via DEV Community

If you've ever opened your browser's DevTools, right-clicked a request in the Network tab, and hit Copy as cURL, you already know the good part: you've captured exactly what the browser sent — headers, cookies, auth, body and all.

The annoying part is what comes next. You wanted Python, and instead you're holding a 30-line shell command full of -H flags and escaped quotes.

This post walks through porting a real curl command to requests by hand, one flag at a time, so the mapping is clear. Do it once like this and you'll recognize every piece next time.

The raw cURL

Here's a representative command copied straight out of Chrome:

curl 'https://api.example.com/v2/search?lang=en&page=2' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1aWQiOjQyfQ.sig' \
  -H 'content-type: application/json' \
  -H 'referer: https://app.example.com/dashboard' \
  -H 'user-agent: Mozilla/5.0' \
  -b 'session=abc123; theme=dark' \
  --data-raw '{"q":"nginx","filters":["open"]}'

Let's take it apart.

1. Method and URL

There is no -X flag here, but there is a body (--data-raw). That detail decides the method: --data / --data-raw makes curl use POST by default, unless you explicitly override the method with flags like -X GET. With -G, curl moves the data into the query string instead.

So don't read the missing -X as "GET" — the presence of a body is what matters.

The URL also carries a query string, ?lang=en&page=2. We'll handle that as real parameters in a moment.

import requests

url = "https://api.example.com/v2/search"

2. Headers → a dict

Every -H 'key: value' becomes one entry in a headers dict. The one rule that bites people: split each header on the first colon, not every colon.

Header values can legitimately contain colons. Look at the referer here, whose value is a full URL: https://app.example.com/dashboard. A blanket split on : would mangle it into pieces; split(":", 1) keeps the value intact.

headers = {
    "accept": "application/json",
    "authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1aWQiOjQyfQ.sig",
    "content-type": "application/json",
    "referer": "https://app.example.com/dashboard",
    "user-agent": "Mozilla/5.0",
}

3. Cookies (-b) → their own dict

The -b flag is a semicolon-separated cookie string. Don't jam it into the headers dict as a raw Cookie line — hand it to requests as a proper cookies argument and let the library do the encoding:

cookies = {"session": "abc123", "theme": "dark"}

4. The JSON body: json= vs data=

The content-type is application/json, so the body is JSON. In requests, that means you pass it as json=not data=:

payload = {"q": "nginx", "filters": ["open"]}

Why the distinction matters:

  • json=payload serializes the dict to a JSON string and sets Content-Type: application/json for you when you do not override it.
  • data=payload form-encodes it as q=nginx&filters=open, which a JSON endpoint will reject or misread.

If the original request had been application/x-www-form-urlencoded, you'd flip to data=. Reading the content type tells you which one to use.

5. Query params

You could leave ?lang=en&page=2 glued onto the URL string, but pulling it into a params dict is cleaner and lets requests handle the encoding:

params = {"lang": "en", "page": "2"}

One caution: if the params in the original URL are already percent-encoded — you see %20, %3D, etc. — don't decode them and pass them through params as well, or you'll double-encode. Either keep the encoded values in the URL string, or pass the decoded values through params, but not both.

6. Putting it together

import requests

url = "https://api.example.com/v2/search"

params = {"lang": "en", "page": "2"}

headers = {
    "accept": "application/json",
    "authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1aWQiOjQyfQ.sig",
    "content-type": "application/json",
    "referer": "https://app.example.com/dashboard",
    "user-agent": "Mozilla/5.0",
}

cookies = {"session": "abc123", "theme": "dark"}

payload = {"q": "nginx", "filters": ["open"]}

resp = requests.post(
    url,
    params=params,
    headers=headers,
    cookies=cookies,
    json=payload,
)

resp.raise_for_status()
print(resp.json())

Because json= sets the content type for you when you do not override it, you could often drop the explicit content-type header and get the same request on the wire.

The gotchas that eat an afternoon

A few things that don't show up until they break:

  • Assuming GET because there's no -X. If --data / --data-raw is present, curl uses POST by default unless you override it with -X GET or use -G to move data into the query string.
  • json= vs data=. The single most common mistake. Match it to the content type.
  • Multipart uploads (-F). These map to files=, not data=. Mixing them up gives you a malformed boundary and a baffling 400.
  • Splitting headers on every colon. Split once; values like URLs and timestamps contain colons of their own.
  • Double-encoding query params. Don't both decode the URL and pass the same values through params.
  • Compression headers. requests handles common decompression for you, and brotli support depends on your installed stack, so you can usually drop accept-encoding unless you are testing compression behavior specifically.

When you just want it done

Doing this by hand once is worth it — now you know what each flag maps to. After that, it's mostly tax, especially for requests with twenty-plus headers.

For the repetitive cases I keep a small browser-based converter open: paste the cURL, get requests code back. It runs entirely client-side, so the command — and any live auth tokens in it — never leaves the browser:

curl to Python requests converter

If you're dropping the request into a crawler rather than a one-off script, there's also a low-key Feapder spider mode that emits a ready-to-run spider class.


What's the gnarliest cURL you've had to port to Python by hand? Mine had a multipart body and eleven cookies. Curious what trips other people up most — json= and data=, or something else?