You write a scraper, test your selectors in the browser, and everything looks right. Then you run the spider and get back nothing. This is the most common point of confusion for developers new to web scraping: the browser shows you the data, your scraper does not find it, and the two are looking at completely different things.
The browser executes JavaScript before you see anything on screen, but your scraper, unless you specifically configure it to do otherwise, does not. It sees the raw HTML the server sent before any JavaScript ran, and on a modern web application that raw HTML is often just a shell: a few <div> tags, some <script> elements, and no actual content.
Figuring out whether a page uses JavaScript rendering takes about two minutes and a browser you already have open, and once you know what you are dealing with, the path forward is clear.
The two-minute test
Open the page you want to scrape. Right-click anywhere on the page and select View Page Source: not Inspect, not DevTools, but View Page Source. This shows you the raw HTML the server sent before the browser ran any JavaScript.
Now search that source for a piece of text you can see on the rendered page: a product name, a price, a headline, anything specific.
If you find it, the content is in the HTML and your scraper can extract it without JavaScript rendering. If you do not find it, the content was injected by JavaScript after the page loaded, your scraper will not find it either, and you need a different approach.
That is the entire test. Everything else in this guide is detail.
Understanding why this happens
It helps to understand the three different ways a page can deliver its content, because each one requires a different scraping approach.
Server-side rendering. The server builds the complete HTML page and sends it. When the browser receives it, all the content is already in the markup. Wikipedia works this way, many news sites work this way, and older e-commerce platforms work this way. This is the easiest case for scraping: requests and Beautiful Soup are sufficient.
Client-side rendering. The server sends a minimal HTML shell with almost no content, plus a JavaScript bundle. The JavaScript runs in the browser, fetches data from an API, and builds the DOM dynamically, which means the content never exists in the original HTML. React, Vue, and Angular applications built as single-page applications typically work this way, and they require either browser rendering or finding and calling the underlying API directly.
Hybrid rendering. The server sends an HTML page with some content already in it (enough for search engines and initial paint), and JavaScript then enhances the page by adding more data, enabling interactivity, and loading supplementary content. Many modern e-commerce and content sites work this way, and depending on which data you need, you may or may not need JavaScript rendering.
The View Source test tells you which case you are in.
Confirming it with Python
The manual test is fast, but if you want to confirm programmatically or build a check into your scraping toolchain, a comparison of the raw response against the rendered DOM is definitive.
import requests
from bs4 import BeautifulSoup
def check_for_js_rendering(url: str, search_text: str) -> dict:
"""
Fetch a page with plain requests and check whether expected text is present.
Returns a diagnostic dict.
"""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
response = requests.get(url, headers=headers, timeout=15)
result = {
"url": url,
"status_code": response.status_code,
"content_length": len(response.text),
"search_text": search_text,
"found_in_raw_html": search_text.lower() in response.text.lower(),
"likely_js_rendered": False,
"notes": [],
}
soup = BeautifulSoup(response.text, "lxml")
# Check for signals that suggest client-side rendering
script_tags = soup.find_all("script")
result["script_count"] = len(script_tags)
# Common JS framework root elements
js_roots = soup.select("#root, #app, #__next, #__nuxt, [data-reactroot]")
if js_roots:
result["notes"].append(f"Found JS framework root element: {js_roots[0]}")
result["likely_js_rendered"] = True
# Very little visible text relative to page size is a strong signal
visible_text = soup.get_text(strip=True)
text_ratio = len(visible_text) / max(len(response.text), 1)
result["text_ratio"] = round(text_ratio, 3)
if text_ratio < 0.05 and len(response.text) > 5000:
result["notes"].append(f"Low text ratio ({text_ratio:.1%}) suggests JS-rendered content")
result["likely_js_rendered"] = True
# Explicit not-found confirmation
if not result["found_in_raw_html"]:
result["likely_js_rendered"] = True
result["notes"].append(f"'{search_text}' not found in raw HTML — almost certainly JS rendered")
return result
# Usage
result = check_for_js_rendering(
url="https://example.com/products/headphones",
search_text="Wireless Headphones"
)
print(f"Status: {result['status_code']}")
print(f"Content length: {result['content_length']} bytes")
print(f"Text ratio: {result['text_ratio']:.1%}")
print(f"Found in raw HTML: {result['found_in_raw_html']}")
print(f"Likely JS rendered: {result['likely_js_rendered']}")
for note in result["notes"]:
print(f" -> {note}")
Reading the signals
Even before searching for specific text, certain patterns in the raw HTML tell you what you are dealing with.
Signal one: the HTML is nearly empty
Open View Source and immediately scroll down. A server-rendered page will have recognizable HTML structure: <header>, <main>, product listings, article text, navigation. A client-side rendered page will often look something like this:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>My Store</title>
<link rel="stylesheet" href="/static/css/main.abc123.css">
</head>
<body>
<div id="root"></div>
<script src="/static/js/main.def456.js"></script>
</body>
</html>
A <div id="root"> or <div id="app"> with nothing inside it is the unmistakable signature of a React or Vue single-page application. Everything visible in the browser was injected by JavaScript into that empty div.
from bs4 import BeautifulSoup
import requests
def has_empty_app_root(url: str) -> bool:
response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(response.text, "lxml")
# These are the standard mounting points for major JS frameworks
for selector in ("#root", "#app", "#__next", "#__nuxt", "[data-reactroot]"):
el = soup.select_one(selector)
if el and not el.get_text(strip=True):
print(f"Found empty JS root: {selector}")
return True
return False
Signal two: JavaScript framework fingerprints
Even on hybrid-rendered pages, certain markers identify which JavaScript framework is in use:
def identify_js_framework(html: str) -> list[str]:
"""Identify JavaScript frameworks present on the page."""
frameworks = []
checks = [
("React", ["data-reactroot", "data-reactid", "__REACT_DEVTOOLS"]),
("Next.js", ["__NEXT_DATA__", "_next/static", "__next"]),
("Vue", ["data-v-", "__vue__", "nuxt__", "__NUXT__"]),
("Angular", ["ng-version", "_nghost", "ng-app"]),
("Nuxt", ["__NUXT__", "__nuxt"]),
("Gatsby", ["___gatsby", "__PATH_PREFIX__"]),
("Svelte", ["__svelte", "svelte-"]),
]
for framework, markers in checks:
if any(marker in html for marker in markers):
frameworks.append(framework)
return frameworks
response = requests.get("https://example.com", headers={"User-Agent": "Mozilla/5.0"})
frameworks = identify_js_framework(response.text)
if frameworks:
print(f"JS frameworks detected: {', '.join(frameworks)}")
print("Page likely requires JavaScript rendering for full content")
else:
print("No major JS framework detected — likely server-rendered")
Signal three: the Network tab in DevTools
For any page where you are unsure, the Network tab in browser DevTools gives you a definitive picture. Open DevTools, click the Network tab, and reload the page.
Look at the first request: the one for the HTML document itself. Click it and look at the Response tab. If the response body contains the data you want, you do not need JavaScript rendering. If it contains an empty shell, you do.
While you are there, also filter to Fetch/XHR and check whether the data you want is arriving via an API call in the background, such as a request to /api/products returning JSON. If it is, you may not need browser rendering at all, because you can call that API directly from Python, which is faster and more reliable than rendering the full page. The guide on intercepting XHR and fetch requests covers this in detail.
Signal four: the page works without CSS but breaks without JavaScript
A quick test that sometimes reveals the answer is to open your browser's developer settings, disable JavaScript, and reload the page. If the content disappears or the page shows a "Please enable JavaScript" message, the content is JavaScript-rendered.
What to do about it
Once you have confirmed the page requires JavaScript rendering, you have three paths forward. They are not mutually exclusive: the right choice depends on what the page is doing, how much data you need, and how much complexity you want to manage.
Path one: find and call the API directly
This is always the first thing to try, because when it works it is the best outcome. Many JavaScript-rendered pages load their data from a JSON API, and if you can find that API, you can call it directly from Python and get clean, structured data without running a browser at all.
Open the Network tab in DevTools, filter to Fetch/XHR, reload the page, and look for requests returning JSON that contains the data you want. Right-click any promising request and select Copy, then Copy as cURL.
import requests
# Reproduced from the cURL command copied from DevTools
response = requests.get(
"https://example.com/api/v2/products",
params={"category": "electronics", "page": 1},
headers={
"Accept": "application/json",
"Referer": "https://example.com/products",
"User-Agent": "Mozilla/5.0",
},
)
data = response.json()
products = data["products"]
print(f"Found {len(products)} products via API")
If the API requires authentication tokens that are generated by JavaScript on page load, you can extract them first and then call the API directly:
from playwright.sync_api import sync_playwright
import requests
def get_api_token(url: str) -> str | None:
"""Extract an API token from a JS-rendered page."""
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url)
page.wait_for_load_state("networkidle")
# Try common locations for auth tokens
token = (
page.evaluate("() => window.__INITIAL_STATE__?.auth?.token") or
page.evaluate("() => localStorage.getItem('token')") or
page.get_attribute('meta[name="api-token"]', "content")
)
browser.close()
return token
token = get_api_token("https://example.com/products")
if token:
response = requests.get(
"https://example.com/api/products",
headers={"Authorization": f"Bearer {token}"},
)
products = response.json()
Path two: use Playwright or Selenium to render the page
When the API approach is not viable (because the data is not available via a clean API, or the authentication is too complex to reproduce), use a browser automation library to render the page and scrape the resulting DOM.
Playwright is the recommended choice for new projects, since it is faster than Selenium, has a cleaner async API, and supports Chromium, Firefox, and WebKit.
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
def scrape_with_playwright(url: str) -> list[dict]:
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
# Block images and fonts to speed up page loads
page.route(
"**/*.{png,jpg,jpeg,gif,webp,svg,woff,woff2,ttf,eot}",
lambda route: route.abort()
)
page.goto(url)
# Wait for the content you actually need, not just page load
page.wait_for_selector("article.product", timeout=10000)
# Parse with Beautiful Soup — the HTML is now fully rendered
html = page.content()
browser.close()
soup = BeautifulSoup(html, "lxml")
products = []
for article in soup.select("article.product"):
products.append({
"name": article.select_one(".product-title").get_text(strip=True)
if article.select_one(".product-title") else None,
"price": article.select_one(".price").get_text(strip=True)
if article.select_one(".price") else None,
})
return products
For async crawls:
import asyncio
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup
async def scrape_page(browser, url: str) -> list[dict]:
page = await browser.new_page()
try:
await page.route(
"**/*.{png,jpg,jpeg,gif,webp}",
lambda route: route.abort()
)
await page.goto(url)
await page.wait_for_selector("article.product")
html = await page.content()
soup = BeautifulSoup(html, "lxml")
return [
{
"name": a.select_one(".product-title").get_text(strip=True)
if a.select_one(".product-title") else None,
}
for a in soup.select("article.product")
]
finally:
await page.close()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
urls = [
"https://example.com/products?page=1",
"https://example.com/products?page=2",
]
tasks = [scrape_page(browser, url) for url in urls]
results = await asyncio.gather(*tasks)
await browser.close()
return [item for page_items in results for item in page_items]
products = asyncio.run(main())
Choosing what to wait for
The most common mistake with Playwright is waiting for the wrong thing. page.wait_for_load_state("load") fires when the initial HTML and scripts have loaded, not when the JavaScript has finished rendering content. Use one of these instead:
# Wait for a specific element you need to be present
page.wait_for_selector(".product-listing")
# Wait for network activity to stop (risky — some pages have background polling)
page.wait_for_load_state("networkidle")
# Wait for a specific API response to complete
with page.expect_response("**/api/products**") as resp_info:
page.goto(url)
response = resp_info.value
data = response.json() # intercept the API response directly
# Wait for an element to contain specific text
page.wait_for_function("document.querySelector('.price')?.textContent?.includes('£')")
Path three: use Zyte API
If you are running scrapers at scale, managing a fleet of browser instances is a significant operational burden, since each browser process uses substantial memory (300–500 MB at minimum), headless browsers require careful configuration to avoid detection, and the infrastructure to run them reliably across many concurrent jobs requires real engineering investment.
Zyte API handles browser rendering and bot detection at the infrastructure level. You send a standard HTTP request and get back rendered HTML, with the browser execution, proxy rotation, and fingerprint management handled by the platform.
import requests
response = requests.post(
"https://api.zyte.com/v1/extract",
auth=("YOUR_API_KEY", ""),
json={
"url": "https://example.com/products",
"browserHtml": True, # request browser-rendered HTML
},
)
data = response.json()
rendered_html = data["browserHtml"]
# Parse with Beautiful Soup as normal
from bs4 import BeautifulSoup
soup = BeautifulSoup(rendered_html, "lxml")
products = soup.select("article.product")
print(f"Found {len(products)} products")
In Scrapy, Zyte API integrates via the scrapy-zyte-api package:
# settings.py
ZYTE_API_KEY = "YOUR_API_KEY"
DOWNLOAD_HANDLERS = {
"http": "scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",
"https": "scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",
}
# Spider: request browser-rendered HTML for specific requests
def start_requests(self):
yield scrapy.Request(
"https://example.com/products",
meta={
"zyte_api_automap": True,
"zyte_api": {"browserHtml": True},
},
)
How to decide
Is the data in the raw HTML? Run the View Source test. If yes, use requests and Beautiful Soup or Scrapy and skip browser rendering entirely.
Is the data loaded via a visible API call? Check the Network tab with the Fetch/XHR filter. If yes, call the API directly from Python, since it will be faster and more reliable than any rendering approach.
Is the data accessible via the API but protected by a token that requires JavaScript to generate? Extract the token with a single Playwright page load, then call the API directly for the actual data scraping: one browser load, many API calls.
Do you need to interact with the page (scroll, click, fill forms, navigate tabs) to reveal the data? Use Playwright or Selenium.
Are you running this at scale and do not want to manage browser infrastructure? Use Zyte API.
When rendering is slower than you expect
Browser rendering is ten to twenty times slower than a plain HTTP request and uses significantly more memory, but a few practical adjustments make a meaningful difference.
Block unnecessary resources. Images, fonts, video, and tracking scripts add load time without contributing to the data you need:
page.route(
"**/*.{png,jpg,jpeg,gif,webp,svg,ico,woff,woff2,ttf,mp4,webm}",
lambda route: route.abort()
)
# Also block common tracking and analytics domains
def block_trackers(route):
blocked = ["google-analytics.com", "googletagmanager.com",
"facebook.net", "doubleclick.net", "hotjar.com"]
if any(domain in route.request.url for domain in blocked):
route.abort()
else:
route.continue_()
page.route("**/*", block_trackers)
Reuse browser contexts rather than browser instances. Creating a new browser is expensive; creating a new context within an existing browser is cheap:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
async def scrape(url):
# New context per task — isolated cookies and storage, reuses browser process
context = await browser.new_context()
page = await context.new_page()
await page.goto(url)
html = await page.content()
await context.close()
return html
results = await asyncio.gather(*[scrape(url) for url in urls])
await browser.close()
Wait for specific elements rather than network idle. networkidle waits until there are no active network connections for 500 ms, and many pages never reach this state because they have background analytics pings. Waiting for the specific element you need is faster and more reliable.
Checking your work
Once you have set up rendering, confirm it is actually returning the content you need:
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
def verify_rendering(url: str, expected_text: str) -> bool:
"""Confirm that rendering produces the expected content."""
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url)
page.wait_for_load_state("networkidle")
html = page.content()
browser.close()
soup = BeautifulSoup(html, "lxml")
found = expected_text.lower() in soup.get_text().lower()
print(f"'{expected_text}': {'found ✓' if found else 'not found ✗'}")
return found
verify_rendering("https://example.com/products/headphones", "Wireless Headphones")
Next steps
Once you know a page requires JavaScript rendering and have chosen your approach, the next challenge is often what happens to the data after you extract it, since cleaning, deduplication, and storage work the same regardless of how the HTML was obtained. If you went the API interception route, the guide on intercepting XHR and fetch requests in the browser covers the full workflow from DevTools discovery to paginated API calls in Python. If you are running Playwright at scale inside Scrapy, web scraping dynamic websites with Zyte API covers how to handle rendering and unblocking without managing browser infrastructure yourself.