I Built an AI Studio That Turns 1 Line of Text Into a 5-Minute Short Drama

From idea to 1080p video — AI scriptwriting, storyboarding, multi-voice dubbing, and video compositing in one pipeline.

I spent the past month building Shortify AI, an open-source platform that takes a creative prompt like "a time-traveling maid from ancient China lands in a modern office" and outputs a full short drama episode — script, illustrated storyboard, multi-character voiceover, and a complete 1080p video.

Here's how it works under the hood, the architecture decisions I made, and the full pipeline that ties it together.

The Problem

China's short drama market hit $70B in 2025. These are vertical, fast-paced 1–5 minute episodes — basically TikTok meets TV series. The production pipeline is:

1. Script → human screenwriter (days)
2. Storyboard → illustrator + director (days)
3. Voiceover → recording studio + voice actors (days)
4. Video → editor + effects (days)

Total: 5–10 people, 1–2 weeks per episode.

I wanted to compress this to: 1 person, 5 minutes.

The Architecture

Here's the end-to-end pipeline:

User Input ("穿越到现代的女将军")
       │
       ▼
┌─────────────────────┐
│   AI Scriptwriter   │  ← GLM-4-Flash / DeepSeek / Qwen
│  (characters +      │
│   scenes + shots)   │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Storyboard Images  │  ← Wan2.7-image / CogView-3-Plus
│  (1 image per shot) │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Multi-voice TTS    │  ← iFlytek WebSocket / Edge-TTS
│  (male / female /   │
│   narrator per role)│
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Video Compositing  │  ← FFmpeg (Ken Burns + AI video)
│  (1080p, subtitles, │
│   background music) │
└─────────┬───────────┘
          │
          ▼
    COS Storage + Share URL

Each stage runs independently and can be swapped — more on that below.

Stage 1: AI Scriptwriting

The LLM acts as a screenwriting assistant. Given a creative prompt, it generates a structured script with:

• Characters (name, gender, voice type, appearance)
• Scenes (location, atmosphere, time of day)
• Shots (character, dialogue, narration, camera direction)

The prompt engineering was the hardest part. Early versions produced flat narration-style scripts. The breakthrough was switching to dialogue-centric formatting:

// Simplified prompt structure
const systemPrompt = `
Generate a short drama script in this format:
{
  "characters": [{ "name": "...", "gender": "male|female", "voiceId": "..." }],
  "shots": [
    {
      "shotNumber": 1,
      "character": "...",
      "dialogue": "...",
      "sceneDescription": "...",
      "cameraDirection": "close-up|wide|over-shoulder"
    }
  ]
}
Rules:
- Each shot = one line of dialogue
- Include scene descriptions for every shot
- Mark characters explicitly so we can assign voice models
- Total length: 7-12 shots per episode
`;

API: GLM-4-Flash (free tier), but any OpenAI-compatible LLM works via our model resolver layer.

Stage 2: AI Storyboard Images

For each shot, we generate an illustration. The challenge wasn't the image generation itself — it was managing cost and consistency.

The model chain:

Wan2.7-image (DashScope, ~$0.03/image)  → 2K resolution, synchronous
  └── fallback → Wanx-v1 (older, cheaper) → 720p, async polling
    └── fallback → CogView-3-Plus (Zhipu) → fallback with different API

For character consistency across shots, we inject appearance descriptors into every prompt:

function buildAppearancePrompt(shot, characters): string {
  const shotChars = characters.filter(c => shot.character === c.name);
  const appearanceDesc = shotChars
    .map(c => `${c.name}: ${c.appearance}`)
    .join(". ");

  return `${shot.sceneDescription}. ${appearanceDesc}. 
          Cinematic lighting, 16:9 widescreen, photorealistic.`;
}

Images are uploaded to Tencent Cloud COS (private bucket) with signed URLs for secure access.

Stage 3: Multi-Voice Dubbing

This was the most fun to build. The LLM script tells us which character speaks in each shot, and we assign voice IDs accordingly:

const voiceMap: Record<string, VoiceConfig> = {
  "male-lead":    { edgeTTS: "zh-CN-YunxiNeural",  iFlytek: "x4_yehaoyun_oral" },
  "female-lead":  { edgeTTS: "zh-CN-XiaoxiaoNeural", iFlytek: "x4_shisan_oral" },
  "narrator":     { edgeTTS: "zh-CN-YunjianNeural", iFlytek: "x4_yunbai_oral" },
  "child":        { edgeTTS: "zh-CN-XiaoxuanNeural", iFlytek: "x4_yunxiaoyan_oral" },
};

Failure handling: The primary TTS (iFlytek WebSocket) sometimes rate-limits. In that case, the pipeline auto-falls back to Edge-TTS (free, runs locally). The entire voiceover stage runs at ~3 seconds per shot, so a 12-shot episode takes ~36 seconds for all dubbing.

Stage 4: Video Compositing — The Hard Part

This is where most of the engineering effort went. The compositing pipeline:

For each shot:
  1. Generate AI video (Wan2.7-t2v / CogVideoX) → or fallback to Ken Burns
  2. Mix voiceover audio → sync with video
  3. Speed-ramp video to match audio duration
  4. Apply fade-in/fade-out
  5. Generate SRT subtitles

Then:
  6. Concat all shot videos → episode
  7. Burn subtitles into final video
  8. Upload to COS
  9. Generate share URL

Ken Burns Camera Effects

When AI video generation is disabled (or fails), we fall back to static images with camera motion. FFmpeg's zoompan filter creates 10 different effects:

# Example: slow zoom-in with fade
ffmpeg -y -loop 1 -i "image.jpg" -i "audio.mp3" \
  -filter_complex "
    [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,
          pad=1920:1080:(ow-iw)/2:(oh-ih)/2:color=black,
          zoompan=z='min(zoom+0.002,1.8)':d=240:s=1920x1080:fps=24,
          fade=t=in:st=0:d=0.4,
          fade=t=out:st=${(dur-0.4).toFixed(1)}:d=0.4[v];
    [1:a]adelay=1|1[a]
  " -map "[v]" -map "[a]" -c:v libx264 -crf 20 -preset fast -y "output.mp4"

10 camera effects: zoom-in, zoom-out, pan-left, pan-right, pan-up, pan-down, zoom-in-left, zoom-in-right, zoom-out-left, zoom-out-right. Each shot picks one in round-robin, making static images feel dynamic.

The "Never Black" Fallback

AI video generation has a ~10-15% failure rate (API timeouts, rate limits, content filters). The original pipeline simply showed a black screen for failed shots. The fix was a three-tier defense:

Tier 1: AI video generation → 80% success rate
Tier 2: Ken Burns zoompan → ~15% (catches most failures)
Tier 3: Static image + text overlay → 100% reliability

Tier 2 was the most fragile — the zoompan filter chain with complex expressions often failed on edge cases (very long/short audio, special characters in subtitles). Tier 3 uses ffmpeg's drawtext to render the subtitle over a dark gradient background — it literally never fails.

The Full Pipeline Script

We abstracted the entire end-to-end flow into a single CLI command:

npx tsx scripts/full-pipeline.ts

This reads the drama metadata from PostgreSQL, iterates over every episode and shot, runs the 4-stage pipeline, and uploads everything to cloud storage. The same script powers the web app's backend API.

Before (4 AI video generations): ~20 minutes per episode (5 min/shot × 4 shots + 1080p encoding)
After (Ken Burns only): ~2 minutes per episode

What I Learned

1. Reliability beats quality.

Early on, I chased the best AI video models (CogVideoX, Kling, Jimeng). But they all have ~10% failure rates, and a single failed shot ruins the entire episode. Investing in a rock-solid fallback chain that guaranteed no black screens was worth more than a 10% quality bump.

2. Model diversity matters.

No single AI provider covers everything. We use:

• Zhipu GLM for script generation (best Chinese LLM for creative writing)
• DashScope Wan for image/video generation (best price/quality ratio)
• iFlytek for TTS (WebSocket, low latency)
• Tencent COS for storage (CDN edge nodes in China)

The model config system lets users bring their own API keys for any stage.

3. Video compositing is the bottleneck.

AI content generation is getting fast and cheap. The bottleneck is now ffmpeg. A 12-shot episode with Ken Burns processing takes ~3 minutes just for the encoding. If you're building an AI video platform, invest in your compositing pipeline first.

Results

Here's a 5-episode drama generated entirely by the pipeline:

Try It Yourself

The project is open-source under MIT:

git clone https://github.com/ycbing/Shortify-AI.git
cd Shortify-AI
npm install
# Configure .env.local with at least DATABASE_URL and GLM_API_KEY
npm run db:push
npm run dev

Or run the full pipeline directly:

npx tsx scripts/full-pipeline.ts

You only need a GLM API key to get started — everything else has free tiers or falls back gracefully.

What's Next

• Advanced character consistency: IP-Adapter / reference image injection for cross-shot face consistency
• Post-prod editing UI: Drag-and-drop shot reordering, manual image replacement
• Vertical mode (9:16): TikTok/Reels/Shorts format
• Sound effects library: Automatic ambient audio (footsteps, rain, doors)

Built with Next.js 16, FFmpeg, PostgreSQL, and a lot of API calls. Star on GitHub if you found this interesting.