惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

博客园 - Franky
Microsoft Azure Blog
Microsoft Azure Blog
阮一峰的网络日志
阮一峰的网络日志
宝玉的分享
宝玉的分享
量子位
N
Netflix TechBlog - Medium
M
MIT News - Artificial intelligence
GbyAI
GbyAI
Apple Machine Learning Research
Apple Machine Learning Research
博客园_首页
博客园 - 叶小钗
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
酷 壳 – CoolShell
酷 壳 – CoolShell
T
Tailwind CSS Blog
Y
Y Combinator Blog
L
LangChain Blog
The Cloudflare Blog
T
The Blog of Author Tim Ferriss
U
Unit 42
Martin Fowler
Martin Fowler
aimingoo的专栏
aimingoo的专栏
G
Google Developers Blog
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
月光博客
月光博客

SerpApi

Getting Started with Jev: Building a Fact Checker with SerpApi Vessel 0.3: Ruby finally gets its Scrapy crawling framework How to scrape Google Play Store app reviews Scrape YouTube Channel Data with SerpApi How to scrape movie and TV results from the Google Play Store SerpApi Weekly Changelog: Sep 07 - 13, 2026 Introducing MCP Bundle Support for SerpApi SerpApi Weekly Changelog: Aug 31 - Sep 06, 2026 We Have Resolved the Google /goto URL Redirect Rollout How to scrape Zillow Reddit promised its users an open internet. Then it closed the door. SerpApi Weekly Changelog: August 24 - 30, 2026 Best web scraping tools in 2026 Google’s New /goto Redirect URLs: Resolution in Progress How to Scrape Walmart Reviews Results Google tried again. We’re moving to dismiss a second time. SerpApi Weekly Changelog: August 17 - 23, 2026 Introducing SerpApi's New Markdown Output SerpApi Surpasses 1.5 Million Activated User Accounts How a Lead Generation Company Scaled Outreach with SerpApi's Google Maps API SerpApi Weekly Changelog: August 10 - 16, 2026 How to scrape book results from the Google Play Store What is an API? Part 3: Dynamic Data Displays with D&D! We Filed an Amicus Brief in U.S. v. Google How to scrape Google Play Store game information SerpApi Weekly Changelog: August 03 - 09, 2026 Introducing SerpApi Search Tools: Real-Time Web Search for Python AI Agents Measuring World Cup Attention: What the data says about Vozinha SerpApi Weekly Changelog: July 27 - August 02, 2026 How to Scrape Google Maps Autocomplete
Understanding HTTP 429: Too Many Requests
Josef Strzibny · 2026-08-28 · via SerpApi
Blog  /  Web Scraping

The HTTP 429 error means a client has sent more requests than a server allows within a given period. Learn why rate limits exist, how to interpret Retry-After, and how to retry requests without making the problem worse.

4 min read

HTTP 429 Too Many Requests is the status code a server returns when a client exceeds a rate limit for a given URL. This often happens when exhausting API quotas or when scraping public web pages too aggressively. The error doesn't necessarily say that requesting the resource isn't allowed, but rather points out that the server cannot handle more requests like that at the moment. Let's go and understand what this code means exactly and how to make it go away.

HTTP status codes

HTTP status codes tell clients what happened with the request they sent. They are grouped by their first digit into 5 categories. The group tells a client whether a request succeeded, requires another step, or failed:

Range Meaning Common examples
1xx Informational response while processing continues 100 Continue
2xx The request succeeded 200 OK, 201 Created, 204 No Content
3xx The client needs to follow a redirect or use cached content 301 Moved Permanently, 304 Not Modified
4xx The client must change something about the request or its behavior 400 Bad Request, 401 Unauthorized, 404 Not Found, 429 Too Many Requests
5xx The server failed to complete an otherwise valid request 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable

HTTP 429 belongs to the 4xx group because the client is expected to change its behavior by sending fewer requests or waiting before trying again. It does not necessarily mean the request data is malformed or that the server is broken.

A request can be valid and still receive a 429 because it arrived after the client's allowed quota had been exhausted.

HTTP 429: Too Many Requests

You might encounter this error while calling an API, scraping a website, submitting forms, polling for updates, or running several background jobs at once. The request may be perfectly valid. The problem is how frequently the requests are being sent.

A typical response looks like this:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 60

{
  "error": "rate_limit_exceeded",
  "message": "Try again in 60 seconds"
}

The server is telling the client to slow down. Retrying the same request immediately usually extends the problem rather than fixing it.

HTTP 429 versus HTTP 503

Both status codes call for controlled retries. The distinction matters for monitoring as repeated 429 responses usually point to client behavior or plan limits, while widespread 503 responses usually indicate a server-side availability problem.

Here's the main difference between these two error status codes:

  • HTTP 429 means the client has exceeded a limit. The response is usually specific to an API key, account, IP address, or request pattern.
  • HTTP 503, known as Service Unavailable, generally means the service itself cannot handle the request because it is overloaded or undergoing maintenance.

Why servers return HTTP 429

Rate limits protect services from accidental overload, abusive traffic, and unexpectedly expensive workloads. They also help providers distribute limited capacity fairly among users.

Common causes include:

  • Sending too many requests per second or minute
  • Running more concurrent requests than an API plan permits
  • Sharing one API key across too many workers
  • Polling an endpoint more frequently than necessary
  • Retrying failed requests immediately and without a limit
  • Scraping pages faster than a website can reasonably serve them
  • Exceeding a daily or monthly account quota

Not every limit is based on an IP address. A service may limit requests by API key, account, endpoint, user, geographic region, or a combination of these factors.

A 429 response may include a Retry-After header telling the client when it can try again. The value can be a number of seconds:

Retry-After: 60

But it can also be an HTTP date:

Retry-After: Wed, 26 Aug 2026 14:30:00 GMT

Clients should support both formats when possible.

Some APIs also return headers describing the active limit, remaining requests, and reset time. Header names vary between providers, so check the API documentation instead of assuming one universal format.

A 503 response may also contain Retry-After, but reducing one client's request rate might not resolve the underlying outage.

How clients should handle HTTP 429

The correct response to receiving the Too Many Requests status code is to reduce pressure on the server immediately and be more mindful in the future.

Respect Retry-After

Do not retry before the server's requested delay has passed. If several workers receive a 429 together, add a small random delay so they do not all retry at exactly the same moment.

Use exponential backoff and jitter

When Retry-After is not specified, increase the delay after every failure. If that doesn't help, increase it again exponentially. You can also consider adding jitter which adds randomness to each delay. This prevents synchronized clients from creating another traffic spike when the waiting period ends.

Limit retries

A retry loop must have a maximum number of attempts. Permanent account quotas, invalid plans, and strict website limits will not be fixed by retrying forever.

Reduce concurrency

A scraper with many workers can exceed a limit even when each worker appears slow. Use a shared rate limiter or queue so all processes follow the same request budget.

Cache and batch requests

Avoid requesting the same resource repeatedly. Cache responses when freshness requirements allow it, combine requests when an API supports batching, and stop polling when the result is no longer needed.

Authenticate correctly

Anonymous requests often have lower limits than authenticated ones. Confirm that the API key is present, belongs to the expected account, and is not being shared unintentionally across environments.

Retrying a 429 response

Here's a simplified Ruby example that respects a numeric Retry-After value and otherwise falls back to exponential backoff with jitter:

require "faraday"

MAX_RETRIES = 5
attempt = 0

loop do
  response = Faraday.get("https://api.example.com/data")
  break unless response.status == 429

  raise "Rate limit exceeded" if attempt >= MAX_RETRIES

  retry_after = Integer(response.headers["retry-after"], exception: false)
  backoff = [2**attempt, 60].min
  delay = retry_after || backoff

  sleep(delay + rand)
  attempt += 1
end

A production implementation should also handle an HTTP-date Retry-After value, request timeouts, network errors, logging, cancellation, and the API's documented rate-limit headers.

Conclusion

If you are getting HTTP error 429 when using an external API or when scraping public web sites, it's important to stop the requests immediately and retry later. How much later can be determined from the Retry-After header if present. If the header is not present, try exponential backoff with jitter to not overwhelm the servers on the other side.