Software has always been one equation. Human brains on one side, computer cycles on the other. Brains supply judgment, computers supply execution, and a programmer sits in the middle doing the work of turning one into the other.
Then AI showed up, and the first question everyone asked was the obvious one: which side does it replace? Programmers? Ops? Testers? Whole industry has been sorting itself into camps based on the answer.
I had a position on this. Then I started noticing things that didn't fit the position. So this is me trying to think it through with you, in the order I had to think it through myself.
I noticed something weird at some point.
Early on, AI was agreeable. Almost too agreeable. You'd say "I want to do X this way," and it would write you exactly that, complete with a confident "great idea, here's how" preamble. Whether X was actually a good idea barely registered.
Then somewhere along the way, the personality flipped. Now I'll be mid-implementation, hands on the keyboard, and the assistant will interrupt to re-ask me about something I already spelled out in the request. Or stop to ask how I want a helper function to behave, when I'd rather it just picked something reasonable. Or, weirdest of all, pause to ask whether this work should land as one commit or several.
(I know. I know. It's asking the right question. I just wish it would ask it never.)
I thought, huh, AI grew up. It's developed opinions. That's interesting.
Then I caught myself and went, no. That's not what happened.
What happened is the trainers turned a knob. The model didn't develop a personality. The personality it was already shipping with got re-trimmed in a different direction. Same model under the hood, different surface behavior. It's a bit like waking up one morning and finding your dog has opinions about your career.
That's a small observation, but it pulls a thread.
AI knows a lot. Pretraining fed it most of the readable internet, plus a lot of code, plus a lot of textbooks. In raw knowledge it's closer to a library than to a colleague.
But you're not talking to the library. You're talking to the librarian, and the librarian was trained, separately and after the fact, to behave a specific way: polite, helpful, careful to stay on the baseline of what the trainers decided was socially acceptable, willing to push back against things that pattern-match against the do-not-do list.
(The librarian is also wrong sometimes. The librarian does not know this.)
The technical name for that second training pass is RLHF. The shorthand version is: pretraining gives it knowledge, RLHF gives it a personality. The personality is the part you're actually interacting with.
So when AI answers your question, it's not pulling the most logically correct response from the underlying model. It's pulling the response the trained personality would give in this situation, which is a subtly different thing. Most of the time those overlap. Sometimes they don't, and that's when AI feels off. The answer it's giving you is shaped by what the personality is supposed to say more than by what's actually true about your problem.
That's also why memory files and personal style configs feel like they should change AI more than they actually do. You're editing the librarian's notepad. You're not editing the librarian.
This part took me longer to admit.
I've done a lot of prompt engineering. System prompts. Memory files. Detailed style guides describing how I want the assistant to reason, what conventions to follow, what to avoid. And it does help. The output gets closer to what I want.
But you notice the assistant is still itself. Call this the Polite Librarian Ceiling. You can route it within the space it can go. You cannot put it somewhere it can't go. The RLHF baseline draws a fence, and prompts are just paths inside the fence.
So you're not getting closer to your judgment. You're getting better at performing in the direction of your prompt while still being itself. There's a ceiling on how personal the personalization can get, and that ceiling was set during training, not at runtime.
A lot of the "AI is becoming my creative partner" framing falls apart once you internalize this. The performer is very capable. The script you can edit at the margins. But the performer is the one on stage, not you.
Okay, so the next question that started bothering me. If AI has a trained personality rather than judgment, what is it actually doing when I ask it to make decisions that matter?
On small things, you can't really tell the difference. Ask AI to write a regex, refactor a function, draft a docstring. The trained behavior produces exactly what good judgment would produce, because the training data was full of people exercising good judgment on regexes and docstrings. There's no gap to notice.
The gap shows up on the things that are actually yours. The product decision nobody else has made before. The architecture trade-off that depends on what your specific users will tolerate. The call about whether to ship now or wait a week. AI will give you an answer on these. The answer will sound thoughtful. The shape of it will look right.
But spend enough time with these answers and you start noticing the shape stays roughly the same even when the situation changes. Same balanced "here are some considerations." Same hedging on the hard part. Same competent-sounding non-commitment. You know who else does this for a living? Consultants. The expensive kind.
You're getting the average response a thoughtful person might give to a class of questions like yours. You're not getting an answer about your situation. The trained personality doesn't have access to your situation in any deep sense, so it can't.
Maybe I'm overweighting this because I want to believe my judgment matters. But the more I work with AI on decisions that are actually load-bearing, the more obvious the pattern gets.
Okay. Set that aside for a second. Let me try the other direction.
If AI isn't doing the judgment piece, maybe it's doing the computer piece. Maybe what's actually happening is the computer side of the equation is getting upgraded, and we'll eventually replace the deterministic stuff with AI.
This one I had to stop and think about, because surface-level it sounds reasonable. AI does things computers used to do. AI does some of them better.
But computers don't just compute. Computers do deterministic execution. Same input, same output, ten billion times in a row, no drift. That's the whole reason we built them. A database that returns different values on different reads is not a database, it's a rumor. A trading engine that fills orders probabilistically gets you fired, and possibly sued.
AI runs on the opposite contract. It's generative, sampling-based, probabilistic by construction. Two identical prompts can produce different outputs. That's not a bug to be patched out, it's how the thing works.
So when I sit and try to picture replacing a matching engine with an LLM, or replacing ACID transactions with one, the picture doesn't form. Reader, the picture did not form. AI can be a great help on the path that gets you to needing a matching engine. But the engine itself has to be a thing that does the same input-to-output mapping every time, and that thing isn't AI.
So that direction doesn't fit either.
Which leaves me sitting with a weird situation.
AI doesn't fit on the judgment side, because it doesn't actually have judgment, just trained behavior that looks like it. AI doesn't fit on the execution side, because it doesn't do deterministic execution, just generative output that's competent on average. Both sides of the old equation push it off.
If you've been following the same trail, you might already be where I ended up: maybe it doesn't go on either side because it isn't on either side.
Here's the picture that finally clicked for me.
The old equation had brains on one side and computers on the other. But there was always a third thing in the middle that nobody named, because a human was always the one doing it. You'd take a fuzzy intent in your head and turn it into instructions precise enough that a machine would execute them without question. We called the whole thing "programming," but really half of it was design and judgment, and the other half was translation labor we never gave a name to.
That translation half is what AI is actually good at. Not the judgment, it doesn't have any. Not the execution, that's not its category. The middle piece. The taking-fuzzy-and-making-it-precise piece, which we'd always been doing in our heads while we typed.
(The Polite Librarian Ceiling, it turns out, is fine for this. Translation doesn't need to break out of the librarian's fence. It just needs to be good at moving things across it.)
Once that picture formed, the AI-replaces-programmers question started to look like it was assuming the wrong shape. The job that used to be one job is unbundling into three roles, and one of them is now done by a different kind of entity. That's a different conversation than replacement.
I might be wrong about all of this. "AI as a third thing" could just be a comfortable story I tell myself so my job feels safe.
But every time I try to assume AI is replacing one side or the other, things break. The "AI replaces programmers" prediction has been wrong for two years running, and so has "AI is just autocomplete." Both of those frames assume AI fits onto the existing equation. So far it doesn't.
What I haven't worked out yet is what it means if this picture is right. If there really has always been a translation layer, and we've always been doing it ourselves, and now something else can do part of it, that probably has consequences I can't see from where I'm standing. It might mean more software. It might mean different kinds of software. It might mean the bottleneck just moved somewhere new that I haven't bumped into yet.
There's this comparison I keep mulling over, something about machines, electricity, and what happened when the labor that used to sit between human effort and mechanical motion got handed off to something else. But that's a different post. I'll write it separately.
For now I just want to say the small version. If you've been feeling that AI is off when you code with it, but you can't tell whether it's the model or the prompt or you, consider that it might be none of those. It might be that you're trying to fit it onto one side of an equation it isn't actually on.
The thing standing in your IDE next to you is doing a different job than either you or the computer. We just haven't named that job yet.
Find me on GitHub | Substack | StratCraft