I Compiled a Custom Chromium Binary to Make AI Fix Bad UIs — Here's How It Works

Most UI optimization tools work the same way: inject some JavaScript, watch click events, run A/B tests, wait months for results.

I wanted something different. Something that could see the DOM the way the browser actually sees it — at the geometry level — and use that to drive real layout decisions without guesswork.

So I built Headless BAI: a behavioral AI pipeline that compiles a patched Chromium binary, intercepts DOM bounding box data in C++, clusters user behavior with K-Means, and routes layout patches through a Llama-3-8B LLM. The result: 31.42% reduction in UI spatial friction across 1,000+ user sessions.

Here's exactly how it works, layer by layer.

The Problem With JavaScript-Based DOM Inspection

Every existing approach — Puppeteer, Playwright, even Chrome DevTools Protocol — has the same limitation: they're sitting outside the rendering pipeline. When you call getBoundingClientRect() in JS, you're asking the browser to compute and serialize layout data across a process boundary. That costs time, introduces CLS artifacts, and you still don't get the raw geometry the compositor is working with.

I wanted native access. Which meant one thing: patch Chromium itself.

Layer 1: Custom Chromium Binary with C++ DOM Instrumentation

The chromium-patches/ folder contains the core of this — diffs applied directly to Chromium's rendering pipeline.

The patch hooks into the layout engine at the point where bounding boxes are finalized for compositing. Instead of waiting for JS to request this data, we intercept it at the C++ level and serialize it to a shared memory buffer. The result: sub-100ms DOM geometry rehydration with zero CLS (Cumulative Layout Shift).

The compiled binary lives in out/Default/ — a full headless Chromium build with our instrumentation baked in.

This is the layer most people skip. It's painful to set up (Chromium's build system is not forgiving), but it's what makes everything downstream actually meaningful.

Layer 2: sensor.js → FastAPI → Supabase Telemetry Pipeline

Once you have clean geometry data, you need behavioral signal on top of it.

sensor.js is a zero-dependency vanilla JS tracker that captures:

• Mouse movement trajectories
• Scroll depth and velocity
• Click coordinates relative to element bounding boxes
• Hover dwell time per element region

Every event gets timestamped and batched. The FastAPI backend (bai-final-year/backend/) ingests these, validates them against our bot filter, and writes to Supabase with Redis caching in front for burst tolerance.

In practice: 10,800+ behavioral events processed across the test sessions, in real time.

Layer 3: Behavioral Cohort Discovery via K-Means Clustering

Raw events are noisy. Before you can do anything useful, you need to segment users by how they actually move through the UI.

The mass_evaluation/ module runs nightly batch jobs that:

1. Featurize sessions: friction score per zone, scroll abandonment depth, click dispersion radius
2. Run K-Means with WCSS optimization (final WCSS: 23.24, 3 cohorts)
3. Apply EMA weighting to smooth temporal drift in behavior patterns

Three cohorts emerged consistently across sessions: high-friction explorers, direct navigators, and scroll-heavy skimmers. Each one needs a different layout strategy.

Layer 4: Llama-3-8B LLM Layout Routing (T=0.1)

Here's where it gets interesting.

Once cohorts are identified, we don't hardcode layout rules. Instead, we prompt Llama-3-8B with the cluster's behavioral fingerprint and the current DOM geometry, and ask it to generate a JSON layout patch.

Temperature is set to 0.1 — near-deterministic. This is intentional. We're not using the LLM for creativity; we're using it as a structured reasoning engine that can translate "cohort 2 users consistently abandon at the 340px scroll depth near the CTA block" into a concrete spatial adjustment.

Output format:

{
  "target_element": "#hero-cta",
  "adjustment": "translate_y",
  "delta_px": -48,
  "rationale": "High scroll-abandonment cohort loses CTA visibility before natural pause point"
}

Patch generation runs in under 500ms per session cohort.

The Metric: 1D Kinematic Friction Score

Standard UX metrics (bounce rate, time-on-page) are too coarse for layout-level optimization. I needed something that could quantify spatial friction — how much a layout fights the user's natural movement patterns.

The Kinematic Friction Score models user cursor/scroll behavior as a 1D kinematic system. High friction = layout elements that interrupt natural movement trajectories. Low friction = elements aligned with how users are actually moving.

After applying AI-generated layout patches across 1,000+ sessions:

Session-weighted friction reduction: 31.42%

That's not a vanity metric. It's computed per-session, weighted by engagement depth, and aggregated across all three behavioral cohorts.

Tech Stack Summary

What I'd Do Differently

Bot filtering is harder than it looks. Our current filter uses statistical heuristics (event timing distributions, movement entropy). A dedicated ML classifier here would be more robust.

The Chromium build pipeline is brutal. 8+ hour compile times on a standard machine. If I were doing this again, I'd look at intercepting at the DevTools Protocol level first to validate the concept before going full custom binary.

Cohort stability across sessions needs work. K-Means doesn't guarantee stable cluster assignments between runs. Moving to an online clustering approach (something like BIRCH or a streaming K-Means) would make the behavioral model genuinely continuous rather than batch-nightly.

Code

→ github.com/Aksharma127/Headless_BAI_orignal

The repo includes the Chromium patches, full backend, clustering pipeline, and evaluation outputs. Still under active development — the architecture is solid but there's cleanup to do.

If you've worked on browser internals, behavioral analytics, or LLM-driven UI systems, I'd genuinely like to hear what you think.

machinelearning #webdev #cpp #ai