I Trained My Own LLM from Scratch in 2025: What That Viral HN Tutorial Doesn't Tell You About the Real Cost

I was scrolling HN on a Tuesday night when I saw the post: "Train Your Own LLM from Scratch", 241 points, 87 comments, and an energy in the thread I recognized immediately — the same one from "I built my own email server" or "I replaced Docker with bash scripts" threads. A mix of genuine technical enthusiasm and collective wishful thinking.

I saved it. Two days later I opened it, cloned the repo, and started measuring everything.

Not because I think I'm going to out-compete Anthropic from my laptop. But because the question nobody answers honestly in those threads is: how much does this actually cost? Not in the abstract. In dollars, in hours, in opportunity cost.

My thesis is this: training an LLM from scratch in 2025 makes sense in exactly two cases — as a deep learning exercise to truly understand transformer architecture, or when you work in a domain so specific and sensitive that no external model can touch your data. In any other case, you're paying an enormous price to get something that Claude Code or DeepSeek already give you for free or nearly free. And the viral tutorial doesn't tell you that.

The Viral Tutorial: What It Promises vs. What It Delivers

The HN post links to an implementation of a small GPT-style transformer, trained from scratch in pure Python with PyTorch. The code is clean. The comments are solid. The author knows what they're doing.

What it promises, implicitly: "you can do this too."

Technically true. Practically, there's a massive gap between running the code and having something useful.

The tutorial trains a ~10M parameter model on an English text corpus (Shakespeare in the classic Karpathy version; this repo uses something similar). The result is a model that generates syntactically coherent text but with no real semantic understanding. It's an educational demo. It's not a production LLM.

I ran it. Here are the real numbers.

The Real Cost: I Measured It, I Didn't Estimate It

Initial Setup

# Environment I used — documenting exact versions
# Python 3.11.9, PyTorch 2.3.1, CUDA 12.1
# Instance: RunPod, RTX 4090 (24GB VRAM), spot instance

pip install torch==2.3.1 datasets transformers
# The repo has its own dependencies — some conflict with what you already have

I used RunPod because Railway, where I normally live, doesn't have GPUs for heavy training. That's already the first hidden cost: if you want to train something serious, you need to leave your normal infra.

Experiment 1: 10M Parameter Model (the tutorial's)

# Small model config — the original tutorial's
config = {
    "n_embd": 384,       # embedding dimension
    "n_head": 6,         # attention heads
    "n_layer": 6,        # transformer layers
    "block_size": 256,   # maximum context
    "vocab_size": 50257, # GPT-2 tokenizer
    "dropout": 0.1
}
# Total parameters: ~10.7M
# Dataset: ~300MB of processed text

Result:

Training time: 47 minutes on RTX 4090
RunPod cost: USD 0.44/hour × 0.78h = USD 0.34
Model output: generates text that looks like English from a distance

Fine. Thirty-four cents. That's not the problem.

Experiment 2: Scaling to Something Remotely Useful

The problem shows up when you try to scale. A 10M parameter model is useless beyond demonstrating you understand the architecture. For basic reasoning capabilities you need to be in the 1B–7B parameter range at minimum — and that completely changes the equation.

# Cost estimate for a 1B parameter model
# Same setup (RTX 4090, RunPod spot):

# Tokens needed for decent training: ~20B tokens
# Tokens per second on RTX 4090: ~8,000 tokens/sec (optimized batch)
# Estimated time: 20,000,000,000 / 8,000 = 2,500,000 seconds
# In hours: ~694 hours
# In days: ~29 continuous days

# RunPod RTX 4090 spot cost: USD 0.44/hour
# Total estimated cost: 694 × 0.44 = USD 305

echo "And that's assuming the spot instance doesn't interrupt every 3-4 hours"
echo "With real interruptions, multiply by at least 1.4"
# Real estimated cost: ~USD 427

Four hundred and twenty-seven dollars for a 1B model that's going to be worse than Llama 3.2 1B, which is free.

Experiment 3: The Cost the Tutorial Ignores

Compute cost isn't the biggest problem. The biggest problem is data cost and your own time.

# What you need for a model that isn't garbage:
# 1. Curated and clean dataset
# 2. Custom or adapted tokenizer
# 3. Continuous evaluation during training
# 4. Hyperparameter tuning (minimum 3-5 runs)
# 5. Checkpoint management — learned this the hard way

# In my case, the data pipeline took 8 hours
# Just to prepare 300MB of clean text
# That time has a real opportunity cost

# Direct comparison I made:
# - 8 hours preparing data for a 10M model
# - vs 8 hours using Claude Code on my Railway project
# The productivity delta is obscene

This connects directly to what I documented in the post about agentic coding in production with real logs: Claude Code in 8 hours generates working code, tests, documentation, and asks me questions I didn't expect. A 10M model I trained in 8 hours generates text that seems coherent.

Why the Viral Tutorial Creates Wrong Expectations

It's not the tutorial author's fault. The code is good and the educational goal is clear. The problem is the context that gets built around it on HN.

I read all 87 comments. There are three types of responses:

Type 1 — The enthusiasts: "Amazing! Now I'm going to train my own model for [specific use case]."

Type 2 — The pragmatists: "Interesting for learning, but for production use fine-tuning on a base model."

Type 3 — The ones who already did it: "Let me tell you how long before my checkpoints crashed on AWS Spot."

The problem is Type 1 is the majority, and Type 3 comments get buried.

My experience with DeepClaude — combining Claude Code with DeepSeek in my agent loop gave me a clear perspective: even DeepSeek V4, which cost hundreds of millions of dollars to train, doesn't beat Claude in my specific use cases. What do I expect to achieve with a 10M model trained in 47 minutes?

The Gotchas the Tutorial Doesn't Mention

1. The Silent Overfitting Problem

# Training loss dropping nicely... or so it seems
# Epoch 1: loss = 4.23
# Epoch 5: loss = 2.17
# Epoch 10: loss = 1.44
# Epoch 20: loss = 0.89  ← this is where the problem started

# The model was memorizing the training dataset
# Validation loss: 1.91 — barely improved since epoch 5
# Classic overfitting you don't see if you're not watching both curves

# What the tutorial shows: training loss going down beautifully
# What the tutorial doesn't show: comparing with validation loss in real time

It took me 20 minutes to notice because the training loss graph looked gorgeous. Classic.

2. Checkpoints Will Blow Up Your Storage

In my first 47-minute run I generated 11 checkpoints at ~240MB each. That's 2.6GB per experiment. In a real fine-tuning experiment, that multiplies by 10 or 20 easily. The tutorial says nothing about checkpointing strategy.

This reminds me of the backup rabbit hole I documented when migrating from pgbackrest to Barman in production: the technical part of a tutorial always looks clean. The real operational side always has friction nobody documents.

3. The Tokenizer Matters Way More Than It Looks

# The tutorial uses the GPT-2 tokenizer — fine for English
# If you want to train on Spanish text or specific code,
# the tokenizer matters enormously

from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Problem: GPT-2 tokenizer was trained mainly on English
# For Spanish, common words tokenize into 3-4 tokens
# vs 1-2 tokens in models trained with Spanish-aware text

# "arquitectura" → ['arqu', 'ite', 'ctura'] # 3 tokens in GPT-2
# vs 1 token in modern Spanish-aware tokenizers

# This isn't a detail — it directly affects the effective context window
# and training efficiency

In my YAML specs for agents I learned that formatting and tokenization details matter more than you'd intuit at first. With your own LLMs, that gets magnified.

4. Inference Costs Money Too

The tutorial ends when the model is trained. But if you want to actually use that model somewhere, you need inference infrastructure. Serving a 1B model in production requires at least 2–4GB of VRAM depending on quantization. On Railway that doesn't exist natively. On any other provider, that's a fixed monthly cost.

When It Actually Makes Sense (for Real)

I'll be concrete because generic answers bore me:

It makes sense if:

You want to genuinely understand how multi-head attention and positional embeddings work — nothing teaches you like implementing it yourself and watching what breaks when you mess something up
You work in a domain with data that can't leave your infrastructure (medical, legal, compliance) and need a specialized model trained on that private data
You're researching something specific about architecture and need total control over every variable

It doesn't make sense if:

You want to "have your own LLM" as a technical achievement without a concrete use case
You think it'll be cheaper than using the DeepSeek API (spoiler: it won't)
You think a 10M–100M parameter model you trained yourself is going to compete with Llama 3.2 or Phi-3

The anecdote that stuck with me most from this whole week: when I migrated the monorepo from npm to pnpm and install time dropped from 14 minutes to 90 seconds, the team couldn't believe it. That feeling of "I improved something real that affects everyone" — training an LLM from scratch didn't give me that. It gave me a different feeling: having deeply understood something about how the system I use every day actually works. That has value. But it's a different kind of value.

FAQ: What Devs Actually Want to Know Before Starting

How much does it cost in dollars to train an LLM from scratch in 2025?

Brutally dependent on size. A 10M parameter model on RunPod with an RTX 4090 costs under USD 1 in compute. A 1B parameter model can run USD 300–500 on a spot instance, assuming the process doesn't get interrupted. For something at 7B parameters that's remotely competitive, you're talking thousands of dollars and weeks of GPU time. Data costs and your own time always exceed compute costs for small models.

Does fine-tuning make more sense than training from scratch?

Almost always yes. Fine-tuning on Llama 3.2 1B or Phi-3 mini gives you a specialized model at a fraction of the cost — hours instead of weeks, tens of dollars instead of hundreds. The only reason to train from scratch is when you need total control over pre-training data or when the goal is purely educational.

Is the HN tutorial worth it for learning?

Yes, genuinely. The code is clean and the transformer implementation is didactic. What it doesn't give you is context about scale, about what happens when you want to do something useful with the result, or about the operational side of checkpoint management, data pipelines, and continuous evaluation. As a starting point for understanding the architecture, it's excellent.

Can I run it on my local machine without a GPU?

You can run the 10M model on CPU, but it'll take 4–8 hours instead of 47 minutes. For larger models on CPU, the times become impractical fast. If you don't have a GPU, RunPod or Vast.ai with spot instances are the most economical option for experimenting.

Is it worth it if I'm already using Claude Code or DeepSeek in production?

Depends on what you're looking for. If the goal is development productivity, no — Claude Code and DeepSeek will give you more value per dollar spent than any model you can train in a weekend. If the goal is understanding how the models you already use actually work on the inside, yes, absolutely worth it as an exercise.

What about training data? Do I need a massive dataset?

For the basic tutorial, no. The repo works with a few hundred MB of text. The problem is that with that amount of data, the resulting model is only useful as a demo. For a model with minimal general capabilities, recent literature suggests at least 1T tokens — something you're not going to collect and clean in a weekend.

My Take: Technical Ego Has a Market Price

I'll be direct because ambiguous endings annoy me: training an LLM from scratch in 2025 is not the smartest technical investment for most developers. It's a deep comprehension exercise disguised as a productive project, and the viral HN tutorial sells it with an energy that implies more than it delivers.

What I did walk away with is concrete: I now genuinely understand why large models need so much compute for capabilities to emerge. I understand why details like tar behavior in Railway pipelines matter — data formatting and processing details matter at every level of the stack, from backups to training data. And I understand why DeepSeek V4 at USD 0.14 per million input tokens is an economic aberration I still haven't fully processed.

If you want to understand transformers, run it. If you want to build something useful, don't lose sight of the fact that 2025 has 70B parameter models available via API for fractions of a cent. Competing with that from scratch is an ego exercise, not an engineering one.

Original source: Hacker News - Train Your Own LLM from Scratch

This article was originally published on juanchi.dev

推荐订阅源

DEV Community