If you've ever typed "let's think step by step" into ChatGPT and watched the answer quality jump, you've already used chain-of-thought prompting without knowing it. That phrase isn't magic — it's a deliberate technique backed by peer-reviewed research.
What It Is
Chain-of-thought (CoT) prompting instructs an AI model to reason through a problem step by step before delivering its final answer. Instead of predicting a response in one leap, the model generates a sequence of intermediate reasoning steps — the "chain of thought" — that leads to the solution.
Why it works comes down to how language models operate: they predict the next token. Without CoT, a model answering a complex problem must compress all reasoning into a single prediction — it can't "work in its head" the way a human uses scratch paper. CoT changes that. Each intermediate step becomes a scaffold that grounds the next, reducing the compounding error that makes AI unreliable on multi-step tasks.
When a model writes "Step 1: identify the variables" before doing algebra, the context window now contains useful intermediate state — and the next token is predicted against something far more constrained than the raw question. The model is, in effect, using its own output as working memory.
Zero-Shot vs Few-Shot CoT
- Zero-shot CoT needs only a trigger phrase — "Let's think step by step" — no examples. The model reasons from scratch.
- Few-shot CoT provides 2–5 worked examples that demonstrate the reasoning process before the actual question, constraining the pattern more tightly.
Zero-shot is easier to implement; few-shot is more reliable for specialized or high-stakes tasks where the reasoning format matters.
| Dimension | Zero-Shot CoT | Few-Shot CoT |
|---|---|---|
| What you provide | Trigger phrase only | 2–5 worked examples + trigger |
| Prompt length | Short | Longer |
| Reliability | Good for general tasks | Higher for specialized tasks |
| Research basis | Kojima et al. (2022) | Wei et al. (2022) |
| API cost | Low | Higher (examples add tokens) |
A concrete before/after: ask a multi-step apple word problem directly and a model may answer "22 apples." Append "Let's think step by step" and it lays out each operation (sell 1/3, receive a delivery, sell half) and lands on the correct 17. Same model, same question — the trigger alone shifts it into step-by-step mode.
The Research Behind It
CoT was formalized by Wei et al. (2022), "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022, Google Brain). The headline result: on the GSM8K grade-school math benchmark, CoT pushed 540B PaLM from 57% to 74% accuracy — surpassing fine-tuned models.
Two findings matter for practitioners:
- Emergent threshold effect. Below roughly 100B parameters, CoT shows little improvement and can actually hurt by generating confident-sounding wrong chains. Above that threshold, gains are dramatic. This is why CoT matters on GPT-4, Claude 3.5/3.7 Sonnet, and Gemini 1.5/2.0 Pro — not on smaller models.
- Self-consistency amplifies CoT. A follow-up technique (Wang et al., 2022) samples multiple reasoning chains and takes a majority vote — improving reliability on tasks with a single correct answer.
The second key paper is Kojima et al. (2022), "Large Language Models are Zero-Shot Reasoners," which showed "Let's think step by step" alone triggers reasoning behavior even without examples — making CoT practical without a library of worked examples per task.
When to Use It — and When to Skip
Use CoT for multi-step math/logic, planning, debugging, argument analysis, decision-making with trade-offs, and research synthesis.
Skip it for simple factual lookups, short creative writing where flow matters, conversational replies, emotional contexts, and high-volume API calls where token cost compounds.
A rule of thumb: if the task is one you'd solve on scratch paper, CoT will help. If it's one you'd answer off the top of your head, CoT just makes the response longer without improving accuracy. Match the technique to the cognitive demand.
How Many Steps?
You generally don't need to specify a count. "Reason through this step by step" lets the model self-determine depth. For more structure, "reason in numbered steps." For very complex problems, add "If any step involves significant uncertainty, flag it explicitly" — this surfaces model uncertainty instead of hiding it in a confident chain.
For the full guide — six copy-ready CoT patterns (universal zero-shot, math/logic few-shot, decision-making, debugging, argument analysis, research synthesis), worked decision examples, and the complete FAQ with sources — I wrote it up here:
https://my-blog.org/tangents/post/chain-of-thought-prompting-explained