I quite enjoyed this talk from Bryan Cantrill where he discusses the difficult engineering problems they overcame while working on their company Oxide.

Some of the problems they ran into were bugs. But these weren’t any ordinary bugs, they were company-destroying bugs: bugs that, if they couldn’t be fixed, would sink the entire company.

And the difficulty in solving these bugs was that they had no precedent. Any documentation or knowledge they could find around the symptoms of the problem was actively incorrect.

In fact, Bryan says that the team’s breakthroughs on these bugs were solutions that an artificial super intelligence would’ve never suggested because they ran against all known and available reasoning, documentation, and knowledge.

His point being: intelligence isn’t everything. Human values are still incredibly important.

Intelligence alone does not solve problems like [the ones we encountered]. Our ability to solve these problems had nothing to do with our collective intelligence as a team. We’ve got a terrific team, but it’s a lot more than just intelligence. And in particular for these [kinds of] problems, and many like them, we had to summon the elements of our character not our intelligence. Our resilience. Our teamwork. Our rigor. Our optimism. […]

We talk about super intelligence, but is anyone talking about super collaboration or super teamwork? We absolutely needed teamwork [at Oxide].

If human values like curiosity are what led to breakthroughs — not the application of synthetic intelligence — why is there so much emphasis on intelligence these days? Bryan has a curt analysis:

This infatuation with intelligence comes from people who just don’t get outside enough.

He notes how intelligence isn’t everything in a job interview. Like, you don’t hire people by giving out an exam and taking whoever scores highest. You try to suss out other aptitudes. Nobody looks at applicants who lack values like teamwork or optimism and says, “Well, they can’t work with anyone and they’re incredibly unpleasant to be around, but their intelligence is great — let’s hire them!”

Intelligence is great, but it’s not everything.

We do a disservice to our own humanity when we pretend that [AI] can engineer autonomously.

A cogent case for the values of our humanity.

More like this please.