Building adaptive local AI inference for real-world hardware instead of benchmark machines.

Running AI models directly in the browser has improved dramatically over the last few years.

With technologies like:

• WebGPU
• ONNX Runtime Web
• WebAssembly
• quantized transformer models

…it’s now possible to run surprisingly capable AI systems locally without uploading data to the cloud.

But there’s a problem that becomes obvious the moment real users start testing your application:

Real hardware is chaotic.

Some users have:

• gaming GPUs
• integrated graphics
• old laptops with 4 GB RAM
• workstations with 32 threads
• browsers with partially implemented WebGPU support
• thermally constrained mobile CPUs

Most browser AI demos are tested on a single developer machine and assume:

• stable GPU acceleration
• enough memory
• predictable threading behavior
• fast inference backends

Once exposed to real users, many of these applications become unstable, extremely slow, or simply crash.

While building Cowslator — a local-first AI transcription platform — this became one of the biggest engineering challenges.

The illusion of “it works on my machine”

A surprising amount of browser AI software is effectively optimized for:

• one browser
• one GPU
• one RAM configuration
• one backend

This works fine in demos.

It fails in production.

For example:

• a model that runs perfectly on a desktop GPU may completely freeze a low-end laptop
• a WebGPU backend may behave differently across browsers
• memory fragmentation can destroy performance on integrated GPUs
• thread counts that help one CPU may hurt another

The result is a poor user experience:

• browser freezes
• out-of-memory crashes
• fans spinning at maximum speed
• unusable transcription times

This is especially problematic for local AI applications, where the user’s machine is responsible for inference.

Why transcription workloads are difficult

Speech transcription is computationally expensive.

Even quantized Whisper models can consume significant:

RAM
VRAM
CPU bandwidth
GPU compute time

And unlike small text demos, transcription often involves:

• long audio files
• sustained inference
• large token generation
• continuous decoding

Now combine that with browser constraints:

• sandboxing
• memory limits
• varying WebGPU implementations
• WebAssembly overhead
• inconsistent multithreading support

The complexity grows quickly.

The naive solution: fixed inference strategy

The simplest architecture is:

loadOneModel();
runInference();

But this creates major problems.

If the model is too large:

• weaker devices crash

If the model is too small:

• transcription quality suffers unnecessarily on powerful machines

If GPU acceleration fails:

• the entire application may become unusable

This is one reason many browser AI demos feel impressive initially but unreliable in practice.

Building an adaptive inference engine

To solve this problem, I started building an adaptive inference engine for local transcription.

Instead of assuming all devices are similar, the application attempts to understand the user’s hardware environment and dynamically choose:

• inference backend
• model size
• quantization level
• threading configuration

At startup, the engine evaluates:

• available RAM
• CPU thread count
• WebGPU availability
• browser capabilities
• estimated memory limits

Then it selects the most appropriate strategy.

Simplified example:

if (gpuAvailable && ramGB >= 8) {
backend = "onnx-webgpu";
model = "medium-q5";
}
else if (cpuThreads >= 8) {
backend = "whisper-wasm";
model = "base-q5";
}
else {
backend = "whisper-wasm";
model = "tiny-q5";
}

This dramatically improves reliability across heterogeneous hardware.

Why fallback systems matter

One of the biggest lessons from browser AI development is:

GPU acceleration cannot be assumed.

WebGPU support still varies significantly:

• browser implementations differ
• drivers behave inconsistently
• integrated GPUs may have unstable memory behavior

Because of this, fallback systems are essential.

Cowslator currently uses:

• ONNX Runtime Web with WebGPU acceleration when available
• a Whisper.cpp WebAssembly fallback when GPU acceleration is not viable

This allows the application to continue functioning even on weaker systems.

Without fallback systems, many users would simply encounter crashes or unusable performance.

Batch transcription changes the workload entirely

Once adaptive inference became reliable, another feature became practical:

Batch transcription

Instead of uploading a single file, users can upload an entire folder of:

• interviews
• lectures
• podcasts
• voice notes
• documentaries

The application then:

• creates a transcription queue
• processes files sequentially
• adapts inference strategy dynamically
• generates outputs locally

This creates a very different workload profile compared to simple browser demos.

Now the system must handle:

• long-running inference sessions
• memory cleanup between files
• scheduling stability
• sustained thermal pressure
• browser responsiveness over time

In practice, batch transcription became an excellent stress test for adaptive local AI systems.

Why local-first AI matters

Most transcription platforms rely on cloud processing:

• upload audio
• wait for server inference

• download subtitles
  This approach is convenient, but it also introduces:

• privacy concerns

• upload bottlenecks

• subscription costs

• API dependence

Local inference changes the model entirely:

• no upload required
• works offline
• uses local hardware
• predictable scaling

As consumer hardware improves, local AI becomes increasingly practical for workloads that previously required cloud infrastructure.

Final thoughts

Browser AI is reaching an interesting stage.

The technology is now powerful enough to run serious workloads locally, but real-world deployment exposes problems that benchmarks rarely reveal:

• inconsistent hardware
• unstable GPU support
• memory constraints
• heterogeneous performance characteristics

The future of local AI applications may depend less on raw model capability and more on adaptive orchestration:

• selecting the right backend
• choosing the right quantization
• scaling to the available hardware dynamically

In other words:

Local AI applications cannot assume homogeneous hardware anymore.

Adaptive inference is becoming essential.

Cowslator is an ongoing experiment in local-first AI transcription:

• browser-based
• privacy-focused
• adaptive to hardware
• capable of batch transcription entirely offline

As local AI tooling matures, I think we’ll see more applications move away from centralized inference and toward adaptive edge computation running directly on consumer hardware.