How to check request header and body?#

When making requests, we may want to see exactly what are being requested. With requests, it is easy do access the request header and request body:

import requests

url = "http://httpbin.org/post"
payload = {"apple": 10, "pear": [20, 30], "img": "http://example.com/demo.jpg"}
r = requests.post(url, json=payload)

print(f"request headers: {r.request.headers}")
print(f"request body: {r.request.body}")

A sample output is:

request headers: {'User-Agent': 'python-requests/2.19.1', 'Accept-Encoding':
'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length':
'69', 'Content-Type': 'application/json'}
request body: b'{"apple": 10, "pear": [20, 30], "img": "http://example.com/demo.jpg"}'

Ref:

• How to display body and headers before sending POST request?
• How can I see the entire HTTP request that’s being sent by my Python application?

How to encode JSON when using application/x-www-form-urlencoded?#

When making requests using json data in requests.post(url, data=json_dict), the Content-Type is application/x-www-form-urlencoded. How is JSON encoded by requests?

In requests, this is handled by class RequestEncodingMixin, it provides _encode_params() method to convert provided into url-encoded string (JSON is converted to query string and some characters are escaped). Under the hood, it is using urllib.parse.urlencode() to encode the json data.

url = "http://httpbin.org/post"
payload = {"apple": 10, "pear": [20, 30], "img": "http://example.com/demo.jpg"}
r = requests.post(url, data=payload)

print(f"request body: {r.request.body}")

The request body is:

apple=10&pear=20&pear=30&img=http%3A%2F%2Fexample.com%2Fdemo.jpg

We can also encode JSON directly using urllib:

import urllib

payload = {"apple": 10, "pear": [20, 30], "img": "http://example.com/demo.jpg"}
print(urllib.parse.urlencode(payload, doseq=True))

The encoded string is the same.

Ref:

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST
• Python requests module: urlencoding json data

Set up max retries#

When we use requests to request a URL, we may sometimes get “max retried exceed” errors due to various causes such as unstable network or frequent request.

We can add retry features using requests to increase the chance of successful request:

from requests.adapters HTTPAdapter

from requests.packages.urllib3.util.retry import Retry

session = requests.Session()

retry_strategy = Retry(total=2)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)

r = session.get(url)

In the above code, we first set the parameter max_retries for HTTPAdapter, which can be a simple number or a Retry object from the urllib3 package.

Then we change the original adapter used by session by using the mount() method. The first parameter is the URL prefix for a certain adapter: when the URL matches the prefix, the corresponding adapter will be used by requests session.

Ref:

• Max retries exceeded with URL in requests