You may have seen it too. This trend of “I wrote some software to solve a problem. I think it’s pretty great. Does anyone have any feedback?”. Maybe it’s a budget app. Or some company management thingy, tracking sales. Or invoicing.

Maybe you take a look. It looks pretty slick. But then you get a feeling of uncanny valley. It’s just not right. Maybe you can’t even put your finger on it.

I’m not an accountant, so when I see some accounting software do something in a different way, it’s interesting. Why is it that way? What can I learn from the fact that a professional thinks it should be this way?

You already know what’s weird about it, if nothing else because of the title of this post. The software works this way because the LLM wrote it that way. There’s no reason. It’s not even wrong.

How do you give “feedback” on that? My feedback would be that you don’t understand the problem you’re trying to solve, and have shown no sign you intend to understand it, so how could you possibly think you can solve it?

You’re not asking for feedback. You’re asking for someone else to do everything.

A house analogy

Analogies sometimes work, sometimes not. I’m not sure if this one will work.

You have a house with stairs that go nowhere, because “houses have stairs, right?”. You didn’t put the stairs there, and you have no sense for why there should be stairs in any particular place, so sure, why not there?

Contrast this with an actual architect. They know why they put the stairs there. They know why they didn’t put the stairs on the other side of the room. It’s not (just) that they have actual knowledge, it’s also that they actually had to think about “how do you get to upstairs, and what’s the experience of walking up those stairs?”. They had to make many small decisions, each of which required thinking, and required that the choices add up to a coherent whole.

The AI put the stairs somewhere, and you said “sure, stairs”. They’re in the kitchen, but since you only had to OK the stairs, not actively choose which room they go in, it didn’t even occur to you that room selection was a thing.

You don’t need architect credentials to do this right. You could have selected a room. It probably would have been a fine room. Maybe a professional could give you feedback on that choice. But if you don’t make choices then you’re not actually contributing to the thing you supposedly built.

You made no choices, so you don’t understand the problem you supposedly solved.

Jurassic park phrased it well

“If I may… Um, I’ll tell you the problem with the scientific power that you’re using here, it didn’t require any discipline to attain it. You read what others had done and you took the next step. You didn’t earn the knowledge for yourselves, so you don’t take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could, and before you even knew what you had, you patented it, and packaged it, and slapped it on a plastic lunchbox, and now you’re selling it”

Except you didn’t even “read what others had done”. Not even that bare minimum.

Artisanally crafted custom solutions

I use my own email client, written in artisanally hand crafted code, like back in the stone age. It solves a problem for me. I have my own youtube UI. And my own RSS reader. For a while I used my own SSH. I know people who wrote their own shell, and that is their daily driver. I plan to move this blog to my own webserver behind a custom SNI router.

Those can get feedback. I’m not saying they’re great, but even if the feedback is “why would you trust OpenSSL more than OpenSSH?”, it’s useful.

But honestly, if someone wants an email client kind of like mine, but customized to their preferences, then they may be better off asking an LLM to write their personal email client. That was not true 5 years ago. I don’t want to use email threading. Maybe you do. It’d be annoying to support both.

AI can make useful custom stuff

I was having some glitches with my video conference, and wanted to troubleshoot. I couldn’t find a tool that did what I wanted exactly. gping came close, but not quite.

I wanted something that pings regularly and frequently, summarizing these many pings about once a second, to illustrate subsecond network outages. And using Linux ICMP sockets, so that it doesn’t require root.

I spent about as much time looking for the perfect tool as I then spent telling an LLM to write it.

I had it write blipfinder, and it solved my problem. I could have had this tool written for me before my meeting had even ended.

Two relevant points about this:

1. I am, actually, a network expert. I know right from wrong in this space.
2. I am not, however, vouching for this tool. It solved my problem, and it’s amazing that custom throwaway tooling can be generated like this.

Just for fun, I had the LLM write it again. That version was also acceptable. It made some different choices. Which choices were best? Meh, nobody gave it any thought.

Why can you not sell it?

1. You weren’t forced to solve the problem, so you weren’t forced to actually understand the problem. You cannot identify a correct solution. Your software will never be fit for purpose for anybody but you.
2. Anybody who’s seen your software can ask an LLM to write it from scratch for them. If you aren’t adding any value by knowing right from wrong, correct from incorrect, then your potential customers are unable to buy expertise from you.

It can still be the right custom software for you. In fact custom software is a huge win for LLM. But selling that is not.

5 years ago there was no software in this uncanny valley. It was either obvious crap, or it was professional. 5 years ago your potential customers couldn’t replace you and your software with a prompt.

But now? If you sell this slop then you don’t have customers, you have “marks”.