How do we change our interviews now that a key part of the job is working with coding agents? A senior engineer is now expected to manage coding agents to produce shippable products, not to write code. But many interviews still test closed-book problem-solving, the one activity the engineer won’t be doing!

The fix sounds simple: rebuild the interview around the actual work. Instead of asking a candidate to implement a quicksort, give them an ambiguous ticket, a real codebase, and the tools to solve it. Do that, and using AI stops being “cheating.” Using AI becomes the test. It’s easier said than done, but this is the direction we’ve been pushing our own interviews over the past six months.

Rebuilding technical interviews for the agentic era is one of the hardest problems on our plate right now, and we’re going deep on it live.
Join Tony Karrer and a panel of engineering leaders for Hiring Engineers in the Agentic AI Era, a free online mini-conference on Friday, October 9, 2026 (8–10 AM PT / 11 AM–1 PM ET). Can’t make it live? Register anyway and we’ll send you the recording.
Reserve your spot →

We’d like to hear what you’re seeing: Have you changed technical screens? What signal do you still trust and what’s changed? What surprised you when you tried AI-allowed rounds? Reply to this email or grab time on my Calendly if you want to compare notes.

Basically, it’s a measurement problem

AI didn’t break technical interviewing. It exposed how long we’d been measuring a proxy for the job instead of the job itself. Karat’s survey of engineering leaders found that 71% say AI makes it harder to assess technical skills, and their CEO argues the root cause isn’t cheating so much as interviews that haven’t kept up with the job. Candidates can produce correct LeetCode answers or polished take-homes, but struggle to debug a multi-file change or explain why they rejected a plausible diff.

What doesn’t work anymore: puzzles without context

Take-home assignments and automated code tests were never strong signals for senior hires. Nowadays they’re close to useless when everyday tools can one-shot them. Interviewing.io let candidates use ChatGPT on modified LeetCode problems. Pass rates jumped sharply and interviewers rarely noticed. Pragmatic Engineer’s reporting on hiring managers found startups dropping take-homes and algorithmic screens for that reason, while Big Tech interviewers report asking harder variants of the same questions instead.

Making the questions harder buys time, not relevance. One hiring manager described re-running their interview questions through the newest model every few months. Eventually it one-shotted their second-round question, and there was no harder one left to reach for. Even with one in hand, a puzzle the model can solve tells you little about how a candidate exercises judgment under real constraints. Templated system design interviews have the same flaw. Candidates learn a script that fits a sixty-minute slot, and the script rarely resembles how your team actually decides tradeoffs.

What survives: judgment in realistic conditions

The interviews that still produce signal look more like the job. Startups and some larger product companies are adding rounds where candidates use AI on a realistic task while an engineer watches how they work. Shopify’s engineering leadership has been public about embracing that model: candidates who don’t use an agent often lose to those who use one well, and the interesting part is what happens next. Do they spot garbage output? Fix an obvious one-line bug by hand or keep prompting? Explain tradeoffs when the model suggests the wrong abstraction?

Strong candidates treat generated code as a draft, validate edge cases, and stay coherent when follow-ups go off script. Weak candidates paste, ramble, or collapse when asked to defend a design choice without a fresh prompt. The shift is to make these skills the whole interview, not just a bonus round after a puzzle.

Two industries pretending to be one

Big Tech and many enterprises are largely fortifying the old process: screen sharing, cheat suspicion flags, harder algorithmic questions, and talk of returning to in-person screens. Startups are redesigning it: dropping take-homes, allowing Cursor or Copilot, or running paid trial days where AI use is expected. Both are rational responses to the same shock, but they optimize for different risks. Incumbents fear false positives from hidden assistance. Fast-moving product companies fear false negatives from banning the environment their engineers already use.

Leaders should name which failure mode they are actually solving. Prohibiting AI without redesigning questions often pushes assistance underground and rewards performers who are mostly good at looking honest on camera. Allowing AI without new rubrics and interviewer training just makes the old false positives easier to generate. Pedigree screening rises when resume signal collapses, which is understandable and still a bad long-term strategy for teams that need builders who can navigate messy codebases, not just familiar logos on a CV.

Recruiter and engineering misalignment makes the signal problem worse before a candidate ever reaches a technical round. When screening criteria are vague, recruiters optimize volume while engineers want depth. Promising profiles never reach the right conversation, or weak ones burn panel time on puzzles that AI already solves. The corrective move is shared ownership of the bar: senior ICs define what “honest talent worth a coding assessment” looks like in practice, recruiters screen to that definition, and both sides treat the early filter as a trust problem as much as a skills problem.

What we’re currently doing

Our own hiring practice has changed multiple times over the past year, and we expect it to change again. (What follows is the coding track, which every engineering candidate goes through regardless of level. Candidates for higher-level roles also get a separate architecture and system design interview.) Our first step is a background interview: a deep dive into work the candidate has done that has real technical complexity, plus a frank discussion of the kinds of challenges that come with the target environment.

Candidates who clear this first step go on to three exercises that mirror how the work actually looks.

• Code analysis. We ask candidates to reason about somewhat obfuscated or obtuse legacy code and explain what it does. That rewards reading, tracing, and judgment, all critical skills in reviewing code that AI is creating.
• A function body against a flimsy specification. It is a simple problem the candidate will encounter often, but the spec is incomplete on purpose. We watch whether they ask the interviewer the right clarifying questions, and then we watch them build the simple thing once the ask is clear. Communication under incomplete information is the skill. Clever algorithm recognition is the LLM’s job.
• An AI-practices walkthrough, framed as day one on a project. The candidate has just cloned a codebase that has never been touched by an AI agent, and their first job is to explore it and get it ready for agentic development. We leave “ready” undefined on purpose. How they approach this task tells us how they think about setting an agent up to succeed. Next we hand them a ticket and ask how they would implement it with AI. Finally we show them the agent’s output and ask how they’d review it.

On that third exercise, we care how they work throughout: Do they understand AI basics and deeper SDLC habits? Do they explore the repo for docs and artifacts? Ask about existing tests, practices, and tech before prompting for code? How do they prompt to implement the ticket, and do they explore the generated code in depth with a critical eye? That last question is where judgment shows up, because the generated code looks plausible and the test is whether they can tell what’s wrong with it.

None of this is a finished template. It is our current bet on measuring the job we actually hire for: reading code, clarifying weak requirements, turning agentic assistance into trustworthy outcomes, and codifying what the agent gets wrong into rules so it doesn’t happen again.

AI is not ending technical interviewing. It is forcing honesty about what we have been optimizing for. The processes that survive will measure how engineers operate with assistance, incomplete information, and accountability for outcomes. That is closer to a PR review than a whiteboard puzzle, and for most product engineering teams, that is the job.

Reading list

• Engineering Interviews in 2026: 3 Trends Hiring Leaders Must Prepare For (Karat, January 2026) — The source of the 71% figure, drawn from 400 engineering leaders across the US, India, and China. The more alarming number is buried further down: confidence that the right candidates are getting offers fell from 68% to 47% in a single year.
• Three Ways AI is Reshaping Traditional Technical Interviews in 2026 (IEEE-USA InSight, 2026) — Where Karat CEO Mo Bhende makes the relevance argument directly: the fundamental job changed, technical interviews did not. Short, and the cleanest statement of the mismeasurement thesis we have found.
• How hard is it to cheat with ChatGPT in technical interviews? We ran an experiment. (interviewing.io) — Thirty-seven interviews where candidates were told to use ChatGPT and interviewers were told nothing. Pass rates hit 73% on verbatim LeetCode and 67% on modified versions, dropping to 25% on fully custom questions. Not one interviewer suspected anything.
• Tech hiring: is this an inflection point? (The Pragmatic Engineer) — Interviews with engineering managers at startups and mid-sized companies, most of whom say hiring is harder now than it has ever been. Useful for the practitioner quotes on why coding problems and take-homes both stopped working at the same time.
• How AI is changing software engineering at Shopify (The Pragmatic Engineer, with CTO Farhan Thawar) — The source for the Shopify approach described above. Thawar shadows interviews himself and uses generated code as a probe, asking the candidate whether it is any good. Worth reading for how differently that round is scored.