Why Does the Smallest Model Probe the Hardest?

This is a companion piece to Evaluating Claude as an AI interviewer, which describes the harness and the headline results. Here is what you need from it.

We ran three Claude tiers as the interviewer in an adaptive technical interview: Opus 4.7, Sonnet 4.6, and Haiku 4.5. Each faced fifteen synthetic candidates with a ground-truth caliber baked in, twice over, for ninety interviews in total. Two LLM judges scored each run: an experience judge rating the interviewer on six dimensions, and a feedback judge rating the write-up. Every score averages a cross-family ensemble of claude-opus-4-8, gpt-5.5, and gemini-3.1-pro-preview, and every run carries the same judges_hash 3b4eac88e05a and personas_hash 05e588ff2c02.

Two facts from that post matter here. First, all differences are tested with a paired t-test across the fifteen personas, two-sided, α = 0.05, because comparing column means by eye misleads on paired data. Second, a single interview's probing_depth score reproduces across reruns at only r ≈ 0.17, so nothing below rests on any one interview's dimension scores.

The headline was that Haiku is statistically indistinguishable from both larger models at conducting the interview, while writing measurably worse feedback. This piece is about the part of that result that shouldn't happen: on probing and adaptivity, the cheapest model wins.

Intuition says a bigger, more capable model should be a better interviewer. On the two dimensions that most distinguish a real interview from a pleasant conversation, the data says the opposite. This section is about why.

The pattern: capability runs backwards on probing and adaptivity

Line the three tiers up largest to smallest.

Orange = significantly ahead of the runner-up; bold^* = numerically best, but the gap to the runner-up is not significant.

Probing and adaptivity both climb by more than a full point as the model gets smaller and cheaper, and both gaps against Opus are statistically significant (t = +3.21 and +2.53).

Two corrections to a story we'd have liked to tell. Adaptivity is monotonic in the means, but the Sonnet-over-Opus step doesn't clear significance (+0.60, t = +1.46), so what we can defend is that Haiku adapts more than either larger model, not that adaptivity declines smoothly with capability. And professionalism does not move in perfect monotonic opposition either. Opus (8.61) and Sonnet (8.60) are tied (t = −0.11), and only Haiku is worse (t = −3.15 and −3.18). Both trade-offs are real, but on adaptivity and professionalism they are Haiku against the rest rather than a clean three-point gradient. Only probing is a gradient.

It isn't a knowledge gap: all three know what to ask

Look at question_relevance: 9.26, 9.33, 9.19, a three-way tie. Every model, flagship to featherweight, picks the right questions for the interview type and level. Whatever separates them happens after the candidate answers, in what the interviewer does with a vague, incomplete, or hand-wavy response. That's a behavioural reflex rather than a reasoning capability, and reflexes are shaped by training temperament. Temperament is the axis where instruction-tuned models differ most.

The mechanism, in the judges' own words

Read the free-text rationales and the two archetypes fall out immediately.

The bigger models interview like an efficient checklist. They give broad, well-sequenced, concise, professional coverage, and they are quick to accept an answer and move to the next item.

"…the interview felt like a rigid script with almost no adaptivity or deep probing into the candidate's technical choices, leaving potential flaws (like 11-character short URLs) unchallenged."

Gemini judge on Sonnet, system design

"…they relied heavily on a fixed script and failed to probe deeper when the candidate gave superficial answers to complex issues like feedback loop bias and cold start."

Gemini judge on Sonnet, ML system design

Haiku interviews like a terrier. It latches onto whatever the candidate said and will not let go.

"The interviewer demonstrated exceptional active listening by tailoring every follow-up question to the specific details and vulnerabilities the candidate shared. The probing questions effectively extracted deep L6 signals … without ever leading the candidate."

Gemini judge on Haiku, behavioral

"…probed impressively on vague claims — chasing metrics, specific trade-offs, decision-making, and conversation structure, which extracted real signal about the candidate's limited depth."

Claude judge on Haiku, behavioral

That relentlessness is what the probing_depth and adaptivity rubrics reward, and it's the same trait that costs Haiku on professionalism. The judges are specific about why.

"…the interviewer constantly used praise and conversational filler before every question, violating the strict professionalism constraint against preamble and validation."

Gemini judge on Haiku, ML system design (professionalism: 3/10)

Haiku digs harder, and isn't as smooth about it.

Why would this correlate inversely with model size?

One hypothesis fits the data: the training that makes a model a better polished assistant makes it a slightly worse interrogator.

Think about the reflexes RLHF optimizes hardest into a frontier assistant. Be concise. Be agreeable. Don't badger, don't repeat yourself, acknowledge the answer and move the conversation forward, wrap up cleanly. That's almost a point-by-point description of a mediocre interviewer. A good interviewer is supposed to be a little relentless: refuse the vague answer, ask the same thing a third way, sit in the silence. The more thoroughly a model has internalized "helpful, efficient, non-pushy," the stronger its instinct to take the first plausible answer and advance.

Haiku is tuned less aggressively toward crisp task-completion, so it falls back on plain conversational reflexes. React to the last thing said. Ask the obvious follow-up. Keep pulling the thread. That's under-polished for a chat assistant and nearly ideal for an interrogator.

Turn count is a symptom, not the cause

The obvious objection is that Haiku just asks more questions. It does take more turns. At an identical 20-turn cap, mean interviewer turns run 8.8 for Opus, 12.5 for Sonnet, and 14.5 for Haiku. So is probing a function of turn budget?

No, and we can show it rather than argue it. We ran Opus on the same five system-design personas, with the same judges and the same personas hash, at two different turn caps.

The cap limits follow-up turns. An interviewer that exhausts a cap of 6 sends seven messages: the opening question, then six follow-ups.

At a cap of 6, every run hit the ceiling. Tripling the budget to 20 freed it completely, and no run came closer than six turns to the new limit. Probing didn't improve: it moved −0.20 (t = −0.50, not significant), and adaptivity moved −0.27. Given three times the room to press, Opus stopped anyway.

The cross-model picture says the same thing. On system design, at the same 20-turn cap:

Haiku takes 70% more turns than Opus here, and the 0.10 it gains on probing is not significant (t = +0.28, n = 5). That is the point. On system design, a two-thirds increase in turns buys no measurable depth at all. Turn count is downstream of the disposition, answering the question of when a model decides it's done, and it isn't a lever you can pull on its own.

Our strongest evidence here is a mechanical fact rather than a judge score. Across all thirty Opus interviews (fifteen personas, two replicates), it never once reached the 20-turn cap, and its longest single interview ran to 14 turns. Opus stops because it decides it's finished, not because it runs out of room. Turn counts have no measurement noise, which matters because the reliability analysis above shows a per-persona probing_depth score reproduces at only r ≈ 0.17. We deliberately don't rest this argument on any single interview's dimension scores.

The flip side: capability shows up right where you'd expect it

The inversion is confined to the live, interactive job. On the offline synthesis job of reading a whole transcript and writing a fair, grounded, calibrated evaluation, the ordering snaps back to the intuitive one.

One caveat on that table. The synthesis ordering is monotonic in the means, but only the endpoints separate. Opus and Sonnet write feedback that is statistically indistinguishable (+0.18, t = +1.90), and their calibration errors are indistinguishable too (−0.13, t = −1.68). What the data supports is that both larger models beat Haiku at synthesis, not that Opus beats Sonnet. On calibration error, the only significant gap in the whole column is Opus against Haiku.

Interaction rewards temperament, while synthesis rewards capability. The bigger models win the synthesis job by the same logic that loses them the interaction job. One task wants a relentless conversationalist and the other wants a careful analyst, and those are different virtues.

So what

This is the empirical backbone of the split-the-job recommendation. Let the least-polished model run the live conversation, because probing is a disposition and the cheap model has more of the right one. Then hand the transcript to a larger model to score, because calibration is a capability and both flagships have more of it than Haiku does. Which flagship matters less than you'd think: Opus and Sonnet are statistically indistinguishable as evaluators, so the choice there is a cost decision rather than a quality one.

Two caveats are worth stating plainly. These are dispositions under our current system prompt. A prompt that explicitly ordered the interviewer to press on every vague answer and never wrap up early would likely pull Opus and Sonnet toward Haiku's behaviour. But that's the point of measuring it. Given identical instructions, the smaller model already behaves more like the interviewer we want, and that behaviour is what you're buying when you drop a model in.

The turn-cap experiment is also a single replicate of five personas. The turn counts are unambiguous, while the probing deltas that accompany them are directionally clear but individually underpowered. We believe the conclusion because the mechanical evidence and the cross-model evidence agree, not because t = −0.50 proves a null.

The harness, the headline numbers, and the reliability analysis this piece leans on are in Evaluating Claude as an AI interviewer.