The joy of building from scratch 2019.07.31

KO | EN

This blog doesn’t use a static site generator or framework like Jekyll, Hugo, or Gatsby. I tried a few of them at first, but they gave me too little freedom to customize, so I decided to build the blog myself. Early on I just wrote posts in HTML, but that was so inconvenient that I built a system for writing posts as JSON files. That too was awkward for longer pieces, so I developed a service that converts Markdown files into HTML files. Then I built a tool to compile and deploy the resulting files. In the end, I’d built a static site generator from scratch.

This kind of thing is called yak shaving. The term was coined by Carlin Vieri, a PhD student at the MIT AI Lab, and it refers to doing a chain of related tasks for a single goal until you lose the original purpose entirely and end up on something completely unrelated. An example mentioned on LangDev IRC makes it clear why it’s called yak shaving.

1. I get an axe to chop down a tree.
2. The axe is too dull, so I go looking for a stone to sharpen it.
3. But I hear that a certain village has a really good stone.
4. I get a yak to travel to that village.
5. The yak’s hair is too long, so I start shaving it.

There’s also an example from Seth Godin, the entrepreneur, marketer, and author.

1. “I should wash the car today.”
2. “Oh no, the hose is busted. I’d better buy a new one at Home Depot.”
3. “But Home Depot is on the other side of the Tappan Zee Bridge. I have to go through the toll, so I need an E-ZPass.”
4. “Wait! I think I could borrow my neighbor’s E-ZPass…”
5. “But Bob won’t lend me his E-ZPass until my son returns the pillow he borrowed.”
6. “The pillow has shed so much yak hair that I can’t just give it back. I’ll have to restuff it with yak hair.”
7. And so, just to wash the car, I end up at the zoo shaving a yak.

Both stories came after the term yak shaving was coined; the term itself was actually born somewhere else. Carlin Vieri, who had played hockey late into a Tuesday night, was eating dinner in the middle of the night and watching TV. On TV was the Yak Shaving Day episode of the cartoon The Ren & Stimpy Show. The plot goes like this:

Yak Shaving Day is five days off. Ren and Stimpy decorate the house by hanging a dirty diaper on the wall and pouring coleslaw into their boots. Then they set shaving cream and a razor on the bathroom sink and pray that a shaved yak will fly in on a magic kayak and bring them gifts. That night, the yak emerges from the bathtub drain, shaves, and leaves a gift in the sink before going: the very scum of the cream it used to shave.^[1]

Carlin Vieri found this bizarre. A few days later, while pulling an all-nighter on paperwork (annoying chores like getting an administrator’s permission, setting up a DHL account, and finding a post office), he told a colleague he was yak shaving. He kept using the phrase with people in the lab over the next few months, and the term caught on.^[2] The cartoon is so strange, and has so little to do with software, that its origin doesn’t seem to be widely known.

One mistake engineers (or the managers running engineering teams) often make is “building from scratch.” An engineer might not quite like the off-the-shelf solutions, or might want to prove their own skill. A client or manager might have a mistaken idea of what existing solutions do, or might think those solutions don’t precisely meet the requirements.

Whether it’s production or a toy project, most projects have a limited budget and limited time. Build from scratch and you eventually start shaving a yak, and once you start shaving, there’s no telling where it ends. In the end you give up the original goal. In a case like that, the right move is to find an alternative that satisfies the core of the requirements and to cut the work down as much as possible.

But yak shaving is fun.

Yak shaving can’t help but be fun. Making something that didn’t exist before, finding and solving problems, digging through a chain of knowledge to understand how something works: all of it pulls engineers in. The very idea of building the thing you want yourself is appealing even if you aren’t an engineer. In The Mythical Man-Month, Frederick P. Brooks Jr. gave these reasons why programming is fun:^[3]

1. The sheer joy of making things.
2. The joy of making something useful to other people.
3. The fascination of fashioning a complex, puzzle-like object out of interlocking moving parts, and watching the rules you built into it play out in subtle cycles.
4. The joy of constant learning.
5. The joy of working with a flexible, easily handled medium of expression.

TeX, too, was born from shaving a yak. TeX is a typesetting system created by Stanford professor Donald Knuth; the name covers the whole system that runs the program, including the typesetting language and the compiler that processes it.^[4] Because it makes formulas easy to enter, it’s widely used in the social sciences and in science and engineering. (LaTeX, a set of macros for using TeX more easily, is what most people use.)

For example, write this in TeX syntax:

-b \pm \sqrt{b^2 - 4ac} \over 2a

and it comes out nicely typeset like this:

In 1976, Donald Knuth was preparing the second edition of The Art of Computer Programming, Volume 2: Seminumerical Algorithms. He wanted to use hot type, the same typesetting used in the first edition, but hot type was no longer available. Unhappy with the alternatives, Knuth came across Patrick Winston’s new book, which had been digitally typeset around that time. Inspired by it, he resolved to build his own digital typesetting system and sketched out TeX’s basic features.^[5]

Knuth built the first version of TeX in the SAIL language, then developed and finished it in WEB, a programming language he created himself.^[6] WEB source mixes documentation and code together, and the documentation and code in a WEB file can be extracted into a TeX file and a Pascal file by two programs called Weave and Tangle, respectively. He called this programming paradigm literate programming. Together with Michael Plass, Knuth also devised the Knuth-Plass line-wrapping algorithm, which decides where to break lines in a paragraph. On top of that, he designed Computer Modern, a font for TeX, and created METAFONT, a language for defining vector graphics. He even developed the DVI (Device Independent) format so that TeX could be output without depending on any particular device.^[7]

So, in order to write a book, Donald Knuth created a programming language, a paradigm, an algorithm, tools, and a typeface. TeX took nearly ten years to build, and the book came out that much later. But it wasn’t a wasted effort.

Of course, this is an extreme success story; most yak shaving fails. You’re supposed to stop at the right point,^[8] but once you start shaving, it’s hard to quit, whether because the time you’ve already sunk in feels too precious to waste or because the shaving itself is fun. Or you really do need to see it through, but you end up quitting anyway, either when the thought “what am I even doing right now?” creeps in and your interest fades, or when the resources allotted to the project run dry.

On the other hand, for someone trying to learn, I think yak shaving is remarkably effective. Most CS coursework demands a certain amount of yak shaving regardless of what the professor intended, and sometimes you get more out of digging into the knowledge around an assignment than out of its main instructions. Put the other way around, shaving a yak always teaches you something. If you set out to build a computing system the yak-shaving way, for instance, you have to study everything from Boolean logic to logic circuits, computer architecture, programming languages, and operating systems. Noam Nisan and Shimon Schocken’s The Elements of Computing Systems walks through exactly this. So even if you never reach the end, as long as you learned something while shaving the yak, that alone makes it worthwhile (or so I’d like to believe).

Anyway, yak shaving is fun.