The Problem

E-commerce sites like Amazon, Walmart, and Target have moved to heavy JavaScript rendering. Traditional HTTP clients (curl, requests, fetch) return empty HTML shells - no product data, no prices, no reviews.

You need a headless browser to render the JS. But headless browsers are slow, expensive, and easy to detect.

Common Anti-Bot Detection Signals

Modern anti-bot systems check for:

• Missing User-Agent or non-standard headers
• Headless Chrome flags (navigator.webdriver === true)
• Abnormal mouse movement patterns
• Request frequency (too fast = bot)
• IP reputation (data center IPs are often flagged)

Enter XCrawl

XCrawl is a web scraping proxy API that handles all of this automatically:

• JS Rendering - Every request goes through a real browser engine
• Proxy Rotation - Residential and datacenter proxies, auto-rotated
• CAPTCHA Bypass - Automatic solving for common CAPTCHAs
• Sticky Sessions - Keep the same IP for multi-page crawls

Quick Demo: Scrape a Product Page

Here's how you'd scrape an Amazon product page with the xcrawl-scraper Node.js SDK:

import { XCrawl } from 'xcrawl-scraper'

const client = new XCrawl({
  apiKey: 'your-api-key'
})

const result = await client.scrapeMarkdown({
  url: 'https://www.amazon.com/dp/B0B1N1L78J',
  proxyLocation: 'us',        // Use US residential IP
  waitForSelector: '#productTitle'  // Wait for content to load
})

console.log(result.content)

Why This Matters

Without these features, you'd need to:

1. Spin up a Puppeteer/Playwright instance
2. Manage a proxy pool yourself
3. Handle CAPTCHAs manually
4. Deal with IP blocks and rate limits

XCrawl bundles all of this into a single API call. One endpoint, one SDK, done.

Pricing

• Free: 1000 credits (enough for testing)
• Hobby: $8/mo for 50K credits
• Starter: $49/mo for 500K credits
• Pro: $199/mo for 3M credits

Try It Yourself

1. Sign up at dash.xcrawl.com (free tier available)
2. Grab your API key
3. Run the code above

Full documentation and SDK: github.com/yanxvdong123/xcrawl-scraper

I built this SDK and API. Questions? Drop a comment below.