Terry Godier has a post titled “Mea culpa” (which, given his framing, might’ve been better titled “Claude’s Culpa”). I’m not sure how much to even trust anything in his post given the backstory, but this line stood out:

I was careless in relying on AI [...] without doing the work to understand

Let’s face it: carelessness is the grain of AI. It’s what the tool encourages and makes easy by default.

Without constant vigilance and deliberate, active participation to cut against this grain — to maintain an understanding — careless outcomes are the default fruit of AI.

And given how good humans are at being constantly vigilant and deliberate, it seems reasonable to expect more stories like this one.

To analogize, AI is pitched like having your very own fruit tree. “Plant a seed, and soon enough, boom! Yummy, juicy fruit for you! So fast and easy!”

But, like most things, it’s not that easy.

Getting good fruit requires all kinds of effort: planting, fertilizing, watering, pruning, thinning, cutting out disease, etc.

Good fruit requires cultivation and care.

Otherwise you’ll just end up with something that looks like a peach, but when you (or someone else) bites into it, you realize it’s disgusting and barely edible.