At PCI Oasis Inc , I was handed a task that sounded simple on paper:

"Help build a crawler that navigates e-commerce websites from the homepage to the checkout page."

Easy enough, right? Open a browser, click some buttons, reach checkout. Done.

Except the internet doesn't want you to do that.

The Problem Nobody Talks About

Every major e-commerce platform, your favourite fashion brands, electronics stores, and sneaker sites run some form of bot detection. Cloudflare. DataDome. PerimeterX. Akamai. Kasada.

These systems are sophisticated. They don't just check if you're sending the right HTTP headers. They watch how you behave in the browser.

They measure things like:

• Does your mouse move in a straight line?
• Do you type at a perfectly constant speed?
• Does your browser have a Canvas fingerprint they've seen a thousand times before?
• Is your WebGL renderer showing signs of a headless cloud VM?
• If anything looks off and I mean anything you get a CAPTCHA, a silent redirect, or just an empty page.

Our crawler had to get through all of that. Autonomously. On any site. Without a human in the loop.

Fingerprint, Meet Counterfeit

Here's what I didn't expect: your browser has a fingerprint, and headless browsers have a very obvious one.

When Chrome runs in headless mode on a cloud server, several things give it away.
The fix? Patch every one of these before the page even loads.

But that's just the beginning. The really interesting stuff is the Canvas fingerprint.

Why Your Browser's Art Class Betrays You

Here's something wild: websites can tell a lot about your browser by asking it to draw something.

The HTML5 Canvas API lets JavaScript render graphics. But the exact pixel output of that rendering varies slightly between real hardware, operating systems, and GPU drivers. Headless Chrome on a cloud VM produces a consistent, identifiable hash — because it always runs on the same virtual GPU.

Bot detection systems have a database of these hashes. If yours matches a known headless browser fingerprint blocked.

The solution? Add noise to the Canvas output. Tiny, imperceptible random variations that make each session produce a unique hash.

Same goes for WebGL — the GPU fingerprint. Headless Chrome on GCP returns "Google SwiftShader" as the renderer. That's a dead giveaway.

The Mouse Problem

This one is my favourite.

Humans don't move their mouse in straight lines. Watch yourself right now your cursor curves, accelerates, decelerates, overshoots slightly and corrects. It's a beautiful, messy, organic path.

Bots move in straight lines. Or they teleport. Both are instant flags.

The solution: Bézier curve mouse simulation.

A Bézier curve is a mathematical curve defined by control points. By generating random control points between the current cursor position and the target, you get a realistic, curved path with natural acceleration and deceleration.

Typing works the same way. Real people don't type at exactly 120ms per keystroke. They have rhythm, occasional hesitation, and natural variance. Gaussian-distributed delays simulate that.

But Wait!! What About the AI Part?

Here's the thing I learned that I didn't expect going in:

The hardest part of building an AI-powered crawler isn't the AI.

It's getting the browser to a state where the AI's decisions can actually execute.

Once you've dealt with fingerprinting, WAFs, and cookie consent banners the AI's job of "figure out how to navigate this checkout" is almost the easy part. The browser is finally in a clean, unblocked state where clicks actually work.

I can't share everything about how the AI navigation works that's the core product. But I'll say this: the most interesting design decision was figuring out when NOT to use AI.

Calling an LLM for every single navigation step is slow and expensive. The real insight was building a system that handles ~60% of decisions with zero AI at all, pure pattern matching and saves the AI for the genuinely hard cases.

That's the architectural principle I'm taking with me: AI is most powerful when it's used precisely, not constantly.

What I Took Away

The impressive part isn't the model. It's everything around the model the infrastructure that gets it into a position where it can actually do something useful.

The browser stealth work, the WAF bypass strategies, the Bézier mouse simulation none of that involves a single API call to an LLM. But without it, the AI is completely useless.

That gap between "AI that works in demos" and "AI that works in production on the real internet" is enormous. And crossing it is mostly an engineering problem, not an AI problem.

If this was interesting to you, the company I worked with PCI Oasis builds security tools for e-commerce payment protection. Their e-skimming labs (the other project I worked on) are open to the public at labs.pcioasis.com if you want to explore real attack simulations in a safe environment.

And if you have questions about any of the techniques above, drop them in the comments. Happy to dig in.

Thanks for reading my article :)