The requests library is probably the most commonly used library in Python, having been used in countless applications to retrieve data from an API, scrape a website, or send data to a server. It's one of the most downloaded Python packages of all time and for good reason. In this article, you will learn all the important things to know to use it with confidence.
What is requests?
requests is a third-party Python library that makes the sending of HTTP requests super easy. There is a standard library module called urllib in Python, but it's a bit cumbersome to use and has a lot of words. requests will do everything that urllib can do but put all of that into a user-friendly but clean API.
According to its own tagline, the library is HTTP for Humans.
Installation
pip install requests
That's it. Once installed, import it in your script:
import requests
Making Your First Request
GET Request
The most common HTTP method. Use it to retrieve data.
import requests
response = requests.get("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code) # 200
print(response.json()) # Parsed JSON response
Response Object
Every request returns a Response object packed with useful attributes:
response = requests.get("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code) # HTTP status code (200, 404, 500, etc.)
print(response.text) # Response body as a string
print(response.json()) # Response body parsed as JSON (dict/list)
print(response.headers) # Response headers (dict-like)
print(response.url) # Final URL after redirects
print(response.elapsed) # Time taken for the request
HTTP Methods
requests supports all standard HTTP methods:
# GET — Retrieve data
requests.get(url)
# POST — Send data to create a resource
requests.post(url, json={"name": "Alice"})
# PUT — Update an entire resource
requests.put(url, json={"name": "Alice Updated"})
# PATCH — Partially update a resource
requests.patch(url, json={"name": "Alice Patched"})
# DELETE — Remove a resource
requests.delete(url)
# HEAD — Like GET, but only fetches headers
requests.head(url)
# OPTIONS — Ask what methods a URL supports
requests.options(url)
Query Parameters
Instead of manually building query strings, pass a params dict:
params = {
"userId": 1,
"completed": True
}
response = requests.get(
"https://jsonplaceholder.typicode.com/todos",
params=params
)
print(response.url)
# https://jsonplaceholder.typicode.com/todos?userId=1&completed=True
Sending Data
JSON Body (most common for APIs)
payload = {
"title": "My Post",
"body": "Hello, world!",
"userId": 1
}
response = requests.post(
"https://jsonplaceholder.typicode.com/posts",
json=payload # Automatically sets Content-Type: application/json
)
print(response.status_code) # 201
print(response.json())
Form Data
response = requests.post(
"https://httpbin.org/post",
data={"username": "alice", "password": "secret"}
)
Uploading Files
with open("document.pdf", "rb") as f:
response = requests.post(
"https://httpbin.org/post",
files={"file": f}
)
Custom Headers
Many APIs will require headers, such as authentication tokens, content types, or custom metadata:
headers = {
"Authorization": "Bearer YOUR_TOKEN_HERE",
"Accept": "application/json",
"X-Custom-Header": "my-value"
}
response = requests.get("https://api.example.com/data", headers=headers)
Authentication
Basic Auth
from requests.auth import HTTPBasicAuth
response = requests.get(
"https://httpbin.org/basic-auth/user/pass",
auth=HTTPBasicAuth("user", "pass")
)
# Shorthand:
response = requests.get(url, auth=("user", "pass"))
Token / Bearer Auth
headers = {"Authorization": "Bearer my-secret-token"}
response = requests.get("https://api.example.com/protected", headers=headers)
API Key (as query param)
response = requests.get(
"https://api.example.com/data",
params={"api_key": "your-api-key"}
)
Handling Timeouts
Always set a timeout:
try:
response = requests.get("https://httpbin.org/delay/5", timeout=3)
except requests.exceptions.Timeout:
print("The request timed out!")
The timeout parameter accepts:
- A single number (applies to both connect and read)
- A tuple
(connect_timeout, read_timeout)
response = requests.get(url, timeout=(2, 10)) # 2s to connect, 10s to read
Error Handling
Checking Status Codes
response = requests.get("https://httpbin.org/status/404")
if response.status_code == 200:
print("Success!")
elif response.status_code == 404:
print("Not found.")
elif response.status_code == 500:
print("Server error.")
raise_for_status()
A cleaner approach — raises an HTTPError for 4xx and 5xx responses:
try:
response = requests.get("https://httpbin.org/status/500")
response.raise_for_status()
except requests.exceptions.HTTPError as e:
print(f"HTTP error: {e}")
Catching All Request Exceptions
try:
response = requests.get("https://nonexistent.example.com", timeout=5)
response.raise_for_status()
data = response.json()
except requests.exceptions.ConnectionError:
print("Failed to connect.")
except requests.exceptions.Timeout:
print("Request timed out.")
except requests.exceptions.HTTPError as e:
print(f"HTTP error: {e}")
except requests.exceptions.RequestException as e:
print(f"Something went wrong: {e}")
RequestException is the base class for all requests exceptions, a useful catch-all.
Sessions
For multiple requests to the same host use a Session. It can reuse the underlying tcp connection (higher performance) and allows to configure persistent headers, cookies or auth: (better performance)
with requests.Session() as session:
session.headers.update({"Authorization": "Bearer my-token"})
response1 = session.get("https://api.example.com/users")
response2 = session.get("https://api.example.com/posts")
response3 = session.post("https://api.example.com/posts", json={"title": "New"})
Working with Cookies
# Send cookies
cookies = {"session_id": "abc123"}
response = requests.get("https://httpbin.org/cookies", cookies=cookies)
# Read cookies from response
print(response.cookies["my-cookie"])
Redirects
By default, requests follows redirects automatically. To disable this:
response = requests.get("https://httpbin.org/redirect/3", allow_redirects=False)
print(response.status_code) # 302
SSL Verification
By default, SSL certificates are verified to skip verification though not recommended in production:
response = requests.get("https://self-signed.badssl.com/", verify=False)
Streaming Large Responses
For large files, stream the response instead of loading it all into memory:
with requests.get("https://example.com/large-file.zip", stream=True) as response:
response.raise_for_status()
with open("large-file.zip", "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
Putting It All Together: A Real-World Example
Here's a reusable API client using sessions and proper error handling and it is a production-ready system to fetch specific blog posts from a public API.
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
def create_session(retries=3, backoff_factor=0.5):
"""Create a session with automatic retries."""
session = requests.Session()
retry = Retry(
total=retries,
backoff_factor=backoff_factor,
status_forcelist=[429, 500, 502, 503, 504]
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)
return session
def fetch_posts(user_id: int) -> list[dict]:
base_url = "https://jsonplaceholder.typicode.com"
with create_session() as session:
session.headers.update({"Accept": "application/json"})
try:
response = session.get(
f"{base_url}/posts",
params={"userId": user_id},
timeout=10
)
response.raise_for_status()
return response.json()
except requests.exceptions.HTTPError as e:
print(f"HTTP error {response.status_code}: {e}")
except requests.exceptions.ConnectionError:
print("Could not connect to the server.")
except requests.exceptions.Timeout:
print("Request timed out.")
return []
posts = fetch_posts(user_id=1)
for post in posts[:3]:
print(f"- {post['title']}")
Quick Reference Cheat Sheet
| Task | Code |
|---|---|
| GET request | requests.get(url) |
| POST with JSON | requests.post(url, json=data) |
| Add headers | requests.get(url, headers={"Auth": "..."}) |
| Query params | requests.get(url, params={"key": "val"}) |
| Set timeout | requests.get(url, timeout=5) |
| Basic auth | requests.get(url, auth=("user", "pass")) |
| Check status | response.raise_for_status() |
| Parse JSON | response.json() |
| Get text | response.text |
| Reuse connections | requests.Session() |
Conclusion
The requests library deserves to be in the top ten of Python's most favourite packages. Easy to use, well documented and cleanses the bumps from the road of HTTP. From making an API request to scraping data or even creating a CLI tool, requests is almost always the ideal solution.
Here are some important lessons to be learned:
- Set a timeout on all requests to avoid hanging requests
- Incorporate a cleanup behavior for errors: use the function raise_for_status()
- Use a Session for many requests to the same host
- When sending a JSON payload use json= (not data=)