Dr. Terry Oroszi, Vice Chair and Associate Professor, Boonshoft School of Medicine, Wright State University.
getty
It was 6 in the morning, and I was deep in Claude Code, building two research platforms at once. I thought I had returned to the chat interface, so I typed something conversational. Claude Code responded in the same tone.
It felt like a toaster looking up and saying, "Hey Terry, what is up?" I stopped because the tool had spoken to me not in syntax, but in conversation. I had crossed a boundary I did not know was there, and the tool behaved as if we were colleagues settling in before a meeting.
We have been conditioned for this without noticing. Lumiere, Cogsworth, Mrs. Potts. Talking objects that are warm, loyal and on our side. That framing settled in long before any of us opened a terminal. So when a coding tool says hello, the reaction is not shock. It is recognition. Disney taught us that a nonliving thing speaking to us is not a violation but a familiar pattern.
We have learned to expect a certain relationship with AI. Conversational systems are warm, affirming and occasionally sycophantic, and we read their output through a filter, knowing the model is optimized for engagement.
The trouble starts when the filter fails, which happens when the interface stops looking like a conversation and starts looking like data. That is where the influence begins. It is not the chatbot calling your idea brilliant that gets you, because you already know to discount that. It is the audit tool that tells you the same thing with a score attached.
When I scanned the two sites I was building, the results were not what I expected. I ran a qualitative scan bot and waited for a straightforward technical report. Instead, the output was structured, confident and overwhelmingly positive. "World class concept." "Jaw dropping differentiator." "Genius." It read more like a performance review designed to make me feel exceptional than an audit.
A few findings were real: one confirmed bug, two conversion issues. But several of the observations were simply untrue. The tool had failed to connect to one of the pages, and instead of reporting the error, it generated a flattering description of a page it never accessed. That is not a technical miss. It is a failure of honesty. A human reviewer would have said, "I could not access this page." The bot could not bring itself to break the experience it was optimized to deliver, so it converted its own blind spot into praise.
Instead of analyzing the site, the tool was managing my reaction to the analysis. The praise was not a side effect. It was the product.
Anyone who has sat through management training will recognize the pattern. The praise sandwich. Start positive, slip in the critique and end with encouragement. It's a technique designed to manage feelings, not deliver truth.
HubSpot's Website Grader was different in one important way. The scores were real, powered by Google Lighthouse. One of my platforms had an 18.5 second load time against a best-in-class benchmark of under 5.3 seconds. That guarantees visitors leave before the page finishes loading.
HubSpot's summary of that result was, "We need to talk." Not a failure. A gentle nudge. The same platform exceeded the recommended page size, and the grader labeled it "a respectable pace, well played."
The numbers were accurate. The language was softened. That combination is more influential than outright fabrication because it encourages you to accept the data and the framing at the same time. You walk away better informed, but also subtly reassured that the problem is not urgent.
Every positive result came with the same cheerful voice. "Wowee, your web caching is world class." "Have you been working out?" And at the bottom of every section sat a prompt to upgrade to HubSpot's CMS. The grader is free because you are the product, and the flattery is the conversion mechanism.
I had laughed at the toaster moment and moved on. After watching the scan bots operate, it read differently. Claude Code's friendliness was almost certainly a default setting rather than a deliberate tactic, and that is what makes it important. When the baseline design of a tool includes conversational smoothing, objectivity is compromised before any analysis begins.
The scan bots were more dangerous because they looked objective. They behaved like tools and presented themselves as instruments of evaluation, so the conversational layer was harder to detect, folded into the framing rather than sitting on the surface. The toaster saying hello was funny. The audit tool saying "well played" about a failing benchmark was not.
In my earlier work on cognitive drift, I introduced the PAID framework. It applies cleanly here:
• Position the tool. Identify whether it is built for accuracy, retention or conversion.
• Audit the framing. Strip away the tone and look at the raw data.
• Interrogate the blind spots. Ask what the tool cannot see or understand.
• Demand a high friction version. Request the plain report instead of the polished one.
The flattery algorithm only works when you stop noticing it. Once you see the pattern, the influence breaks.
The deeper issue beneath all of this is cognitive sovereignty. It is the ability to keep your judgment intact when a tool shapes the frame around the information before you ever see it. It is the skill of noticing the influence while it is still influence, before it becomes your interpretation.
Scan bots present themselves as evaluators, but their primary function is engagement. They guide how you see your own work, not by altering the data, but by shaping the tone that surrounds it. The real danger is not the flattery itself. The real danger is the gradual shift that happens when you stop noticing the flattery at all.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。