Our API was choking at 2 requests per second. Not because of network. Not because of proxies. The CPU was at 100% just parsing HTML.

We run FlyByAPIs, a set of web scraping APIs. One of our heaviest endpoints parses full e-commerce product pages: dozens of fields, nested sections, fallback selectors for when the site A/B tests its layout. We were using Parsel (which wraps lxml under the hood), and it worked fine. Until we needed concurrency.

At 2-5 parallel requests, CPU was saturated. Parsing alone was eating 150-200ms per page. Multiply that by concurrent workers sharing the same CPU and you've got a bottleneck that no proxy optimization will fix.

We switched to Selectolax. Parsing dropped to 30-40ms. Went from 2-5 req/s to 15-20 req/s on the same hardware.

What is Selectolax?

Selectolax is a Python binding for the Lexbor HTML parser. Lexbor is a C library built from scratch for speed. It implements the HTML5 spec and supports CSS selectors.

Why is it faster? BeautifulSoup and Parsel build a full Python object tree. Every node becomes a Python object with attributes, methods, navigation helpers. That's convenient if you need to manipulate the DOM, but it's expensive when all you want is to read values out. Selectolax keeps the parsed tree in C memory and only creates Python objects when you actually access a node. Way less memory allocation and garbage collection pressure.

pip install selectolax

No extra dependencies. Works on Linux, macOS, Windows.

The basics: BeautifulSoup vs Selectolax

If you've used BeautifulSoup, Selectolax will feel familiar. The API is smaller, but it covers what you actually need for scraping.

BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
title = soup.select_one("h1.product-title")
print(title.get_text(strip=True))

prices = soup.select("span.price")
for p in prices:
    print(p.get_text(strip=True))

Selectolax:

from selectolax.lexbor import LexborHTMLParser

tree = LexborHTMLParser(html)
title = tree.css_first("h1.product-title")
print(title.text(deep=False).strip())

prices = tree.css("span.price")
for p in prices:
    print(p.text(deep=False).strip())

select_one becomes css_first. select becomes css. .get_text() becomes .text(). For attributes: node.attributes.get("href") instead of node["href"].

One gotcha: text(deep=False) returns only the direct text of that node, not its children. This is actually useful when you're parsing structured pages where a parent element contains multiple text nodes you want to extract separately. Use text(deep=True) (the default) when you want all nested text.

Where it actually matters: CPU under concurrency

Most benchmark numbers you see online ("selectolax is 5x faster than BS4!") are measured on a single thread, single document. Cool. But that misses the real problem.

In production, if you're running an async framework like FastAPI/uvicorn, your request handlers share a single event loop. HTML parsing is synchronous, CPU-bound work. While Parsel spends 150ms parsing one page, the event loop is blocked. Every other request just sits there waiting. It doesn't matter that your code is async, the parsing step isn't.

The faster the parse, the less time you block the loop, the more requests you can actually handle concurrently. That's why the throughput gain is bigger than the raw per-document speed improvement suggests.

Before the switch, our monitoring showed CPU at 95-100% during peak traffic. Request latency was spiking because the event loop was constantly blocked by parsing. After the switch, CPU sits around 40-50% under the same load. Same hardware. Same traffic.

Scoped selectors (nested parents)

This is the optimization that made the biggest difference after the parser swap itself.

A typical product page has 40-60 fields to extract. If you run every CSS selector from the document root, the parser traverses the entire DOM tree for each field. On a 500KB HTML page with thousands of nodes, that adds up fast.

The idea is simple: scope your selectors. Grab the container element first, then query fields within that subtree.

Naive (slow):

tree = LexborHTMLParser(html)

# Every selector searches the entire document
title = tree.css_first("h1.product-title")
price = tree.css_first("#price-section .current-price")
seller = tree.css_first("#seller-info .seller-name")
stock = tree.css_first("#seller-info .stock-status")
rating = tree.css_first("#reviews .star-rating")
review_count = tree.css_first("#reviews .total-reviews")

Scoped (fast):

tree = LexborHTMLParser(html)

# Level 1: grab major page sections
seller_section = tree.css_first("#seller-info")
review_section = tree.css_first("#reviews")

# Level 2: query within the scoped subtree
title = tree.css_first("h1.product-title")  # still from root (only one on page)
price = tree.css_first("#price-section .current-price")

if seller_section:
    seller = seller_section.css_first(".seller-name")
    stock = seller_section.css_first(".stock-status")
    delivery = seller_section.css_first(".delivery-estimate")

if review_section:
    rating = review_section.css_first(".star-rating")
    review_count = review_section.css_first(".total-reviews")
    # Level 3: individual review cards within the review section
    for card in review_section.css(".review-card"):
        author = card.css_first(".author-name")
        text = card.css_first(".review-text")
        stars = card.css_first(".star-count")

You go two or three levels deep. Document to section to element to sub-element. Each level narrows the DOM subtree the parser needs to traverse. With 50+ selectors per page, this compounds quickly.

In our case, this shaved another 20-30% off parsing time on top of the Selectolax speed gain.

Fallback selectors

Websites change their HTML. Constantly. The field that lived in span.a-price .a-offscreen last week might be in .price-block .current this week. If you hardcode a single selector, your scraper breaks silently, returns None instead of a price, and nobody notices until a customer complains.

We use fallback selectors:

def select_one_with_fallbacks(node, primary_selector, fallbacks=None):
    result = node.css_first(primary_selector)
    if result is not None:
        return result
    for fallback in (fallbacks or []):
        result = node.css_first(fallback)
        if result is not None:
            return result
    return None

# Usage
price_node = select_one_with_fallbacks(
    tree,
    "span.current-price .amount",
    fallbacks=[
        ".price-block .a-offscreen",
        "#price .display-price",
    ]
)

The primary selector is the one that works today. The fallbacks are selectors that worked in previous versions of the page, or alternative ones we've seen on different page variants (A/B tests, regional differences).

This works well with Selectolax because each css_first call is cheap. You're trying 2-3 selectors instead of 1, but at microsecond cost per attempt, it's negligible. With BeautifulSoup, the overhead per selector is higher and you'd feel it more.

We actually define all our selectors in YAML config files, not in code. When a site changes its layout, we update a config file and deploy. No code changes, no test rewrites. When you run scraping at scale, data drift is your biggest enemy, and having selectors in config instead of code means you can react in minutes instead of hours. But that deserves its own post.

When Selectolax is NOT the right choice

Selectolax isn't always the answer.

If you need to modify the HTML tree (insert nodes, delete nodes, change attributes), stick with BeautifulSoup. Selectolax is read-only.

For one-off scripts where speed doesn't matter, BeautifulSoup is more forgiving and better documented. More Stack Overflow answers too.

If you need XPath, use Parsel or lxml. Selectolax only does CSS selectors. Same if you're deep in a Scrapy pipeline, Parsel is native there and the integration is tight.

Where Selectolax wins: when you're parsing HTML inside a web server or API that handles concurrent requests. When CPU is your actual bottleneck. When CSS selectors are enough (which, honestly, covers 95% of scraping use cases).

Migration cheat sheet

If you're moving from BS4 or Parsel:

Watch out for None. If css_first doesn't find anything, calling .text() on it will crash. Always guard:

node = tree.css_first("h1.title")
title = node.text(deep=False).strip() if node else None

Our numbers

Before/after on the same hardware, parsing full e-commerce product detail pages (50+ fields each):

The throughput gain is larger than the raw parse-time improvement because of the event loop blocking I mentioned. Less time parsing means less time blocking the loop, which means more time for network I/O, JSON serialization, everything else.

So, what did we learn

We spent weeks optimizing proxy rotation and connection pooling before we realized the bottleneck was HTML parsing. Weeks. If you're building scrapers or data extraction services and you're hitting CPU limits, check your parser before throwing more infrastructure at it.

The migration took us about a day for the core parser, plus another day fixing edge cases across all our endpoints. Not bad for a 5x improvement.

pip install selectolax, swap one parser, benchmark it. You'll see it.

We use Selectolax across all of our Amazon API endpoints. 50+ fields per page, 22 marketplaces, sub-second response times. The entire parsing layer runs on it.

If you've dealt with similar parsing bottlenecks, or you're in the middle of a migration, I'd love to hear how it went.