I Built an AI 3D Model Generator — Here's How I Handle Me...

I shipped www.imgto3d.ai a few months ago. It's an AI-powered 3D model generator: you upload an image, pick a generator (upscale, denoise, face recovery, or full restoration), and get back a .glb file you can drop into Blender, Unity, or straight into a web page.

The AI backend part — calling Replicate, polling for results, handling credits — was straightforward. The part that actually made me lose sleep? Getting those generated 3D assets to render smoothly inside a Next.js app without turning the user's phone into a hand warmer.

This is the story of how I built the frontend, and the specific performance tricks I had to pull to make browser-based 3D preview not feel like a PowerPoint from 2003.

What the Product Actually Does

ai3dgen.com runs four generators:

Upscale — takes a low-poly preview and bumps the geometry density.
Denoise / Unblur — cleans up noisy AI outputs.
Face Recovery — fixes distorted facial topology on character models.
Restoration — full pipeline for damaged or low-quality inputs.

Users pick their settings, burn some credits, and get an email when the mesh is ready. The critical moment is when they click "Preview" and expect to spin the model around in 3D right there in the browser.

That preview is where everything almost fell apart.

The Problem: AI Doesn't Care About Your Frame Rate

When users want to turn a 2D image into a 3D model online, they expect two things: fast processing and a crisp, interactive preview. The AI pipeline delivers beautiful meshes — thousands of polygons, 4K textures, the works. But it delivers them with zero optimization. There's no LOD. No texture atlasing. No one ran it through a decimation modifier.

My first attempt was embarrassingly naive: dump the raw .glb into a <Canvas> and call it a day. On my M2 MacBook, buttery smooth. On a 2021 Android phone? Tab crash. iOS Safari? "A problem occurred with this webpage so it was reloaded." Every. Single. Time.

I realized I wasn't just building a product — I was building a 3D runtime that had to survive the wild west of consumer hardware.

The Stack

Next.js 14 (App Router) — SSR for SEO, client components for the viewer.
Tailwind CSS — because I'm not writing raw CSS in 2026.
React Three Fiber (R3F) + @react-three/drei — declarative 3D scenes as React components. If you're still wrapping raw Three.js in useEffect, you're making life harder than it needs to be.
Replicate API — the AI pipeline that generates the actual meshes.

R3F makes it easy to get something on screen. It does not make it easy to get something performant on screen. That part is on you.

The Viewer Component (Production Code)

Here's the actual component running in production. Every line in here exists because something broke in production first.

import React, { Suspense, useMemo } from 'react';
import { Canvas } from '@react-three/fiber';
import { OrbitControls, useGLTF, Center, Environment } from '@react-three/drei';

function Model({ url }: { url: string }) {
  // useGLTF caches by URL. But if your URL is a signed S3 link 
  // that changes every refresh, congrats — you just disabled caching.
  const { scene } = useGLTF(url, true);

  // Clone the scene so multiple instances don't share materials
  const clonedScene = useMemo(() => scene.clone(), [scene]);

  return (
    <primitive 
      object={clonedScene} 
      scale={1.5} 
      castShadow 
      receiveShadow 
    />
  );
}

function Loader() {
  return (
    <div className="absolute inset-0 flex flex-col items-center justify-center bg-[#0d0e12] text-white z-10">
      <div className="h-10 w-10 border-2 border-gray-700 border-t-cyan-500 rounded-full animate-spin" />
      <p className="mt-4 text-sm text-gray-400">Decoding mesh...</p>
    </div>
  );
}

export default function Viewer3D({ modelUrl }: { modelUrl: string }) {
  return (
    <div className="relative w-full h-[500px] bg-[#0d0e12] rounded-xl overflow-hidden border border-gray-800">
      <Suspense fallback={<Loader />}>
        <Canvas
          camera={{ position: [0, 1.5, 4], fov: 45 }}
          gl={{ 
            antialias: true, 
            preserveDrawingBuffer: true,
            powerPreference: "high-performance"
          }}
          dpr={[1, 2]} // Clamp pixel ratio — retina screens will murder your FPS otherwise
        >
          <ambientLight intensity={0.6} />
          <directionalLight 
            position={[5, 5, 5]} 
            intensity={1.0} 
            castShadow 
            shadow-mapSize-width={1024}
            shadow-mapSize-height={1024}
          />

          <Center>
            <Model url={modelUrl} />
          </Center>

          <OrbitControls 
            enablePan={false}
            enableZoom={true}
            minDistance={1.5}
            maxDistance={8}
            maxPolarAngle={Math.PI / 1.5} // Prevent going below the ground
            autoRotate
            autoRotateSpeed={1.0}
          />

          <Environment preset="city" />
        </Canvas>
      </Suspense>
    </div>
  );
}

Three Things I Learned the Hard Way

1. Draco Compression Is Non-Negotiable

The AI backend originally served raw GLBs. A typical output was 45MB. On a 3G connection, that's 30+ seconds of staring at a spinner. Users bounce before the first vertex loads.

I added server-side Draco compression. File sizes dropped to ~8MB. Visually identical. The catch? You need to tell useGLTF to use a Draco decoder, and that decoder needs a .wasm file. Host that .wasm on a different domain without CORS headers? Silent failure. The model just never loads, and you get no error in Sentry.

// In your app entry or a useEffect
useGLTF.setDRACOLoader(new DRACOLoader().setDecoderPath('/draco/'));

Also, Draco decoding is CPU-intensive. I throttle it by limiting concurrent loads — if a user rapidly switches between models in the gallery, I cancel the previous request. Otherwise, the main thread chokes and the UI freezes.

2. iOS Safari Is the Final Boss

Apple's WebGL implementation on iOS has a hard memory limit. Not a suggestion — a hard ceiling. Exceed it, and Safari nukes your tab without warning. No exception. No graceful degradation. Just reload.

I discovered this testing a high-poly model with an 8K texture. Chrome desktop? Perfect 60 FPS. iPhone 14? Instant reload.

My fixes:

Cap texture resolution at 2K for mobile (detected via navigator.hardwareConcurrency + touch event sniffing).
Clamp dpr to [1, 1.5] on mobile instead of [1, 2].
Implemented a "Lite Mode" toggle that swaps to a lower-poly version of the mesh. I hate that it exists, but users on older devices love it.

3. Don't Trust the Default Lighting

A 3D model under default lighting looks like a plastic toy from a gas station. I spent way too long tweaking individual lights before realizing R3F's <Environment> component with a preset solves 90% of the problem. It gives you realistic reflections and ambient occlusion for free.

The other 10%? A subtle contact shadow under the model using <ContactShadows> from Drei. It grounds the object and makes it feel like it's sitting on a surface, not floating in a void.

Why This Architecture?

I chose Next.js + R3F because I needed two things that usually fight each other:

SEO — the marketing pages need to rank. Next.js App Router handles that.
Interactive 3D — the viewer needs to be a rich client experience. R3F handles that.

The split is clean: server components for landing pages, metadata, and auth. Client components wrapped in 'use client' only where the Canvas lives. I keep the bundle small by dynamically importing the viewer:

const Viewer3D = dynamic(() => import('@/components/Viewer3D'), { ssr: false });

This way, the Three.js payload only hits users who actually click "Preview." Everyone else gets a fast, lightweight landing page.

Where It Stands Now

ai3dgen.com is live. It handles four distinct generation pipelines, serves 12 languages, and processes thousands of conversions per week. The 3D viewer went from "crash my phone" to "smooth on a 4-year-old mid-range Android."

The hardest part wasn't the AI. It was respecting the user's hardware.

Try It / Break It

If you're building something with in-browser 3D — an AI tool, a product configurator, a portfolio — the ecosystem is mature enough that you don't need a graphics PhD. R3F + Drei + a bit of performance paranoia gets you 90% of the way there.

If you want to stress-test your own 3D viewer, image-to-3d AI will give you some gloriously unoptimized GLBs to throw at it. Upload an image, grab the mesh, and see if your renderer survives.

How are you handling 3D assets in your web apps? Running into the same memory limits, or have you jumped to WebGPU already? Drop a comment — always curious to see how others are solving this.

推荐订阅源

DEV Community