Evaluating Claude as an AI Interviewer

TL;DR

Interview Me runs a live, adaptive technical interview and then writes you a calibrated evaluation. An LLM drives both halves. So we built a harness that grades the interviewer the way we'd grade a candidate, and ran three Claude models through it. Every score below is the averaged verdict of three judges from three different labs: Claude, GPT, and Gemini, because a judge drawn from the same family as the interviewer grades its own house style generously.

Orange marks a value that leads by a statistically significant margin over the runner-up (paired t-test across the fifteen personas, two-sided, α = 0.05). Bold^* marks a value that is numerically best but not significantly ahead of the runner-up. Those rows are ties. A row can still hide a real gap further down. On feedback quality Opus and Sonnet are tied with each other, yet both beat Haiku decisively. See Reading the tables.

Why you have to evaluate an AI interviewer

An AI interviewer has two jobs, and they fail in different ways:

1. Conduct the interview. Ask relevant questions, follow up on vague answers, adapt to the candidate, and not leak the answer. A model that nods along and accepts the first thing it hears produces a pleasant, useless interview.
2. Write the feedback. Turn a transcript into a fair, specific, actionable evaluation with a score and a hire verdict, without hallucinating moments that never happened or going soft.

Vibes don't catch regressions in either. Swap the model, tweak a prompt, raise the turn cap: did probing get better, or did the model just get chattier? Did feedback get fairer, or only more generous? Answering that takes numbers, and the numbers have to mean something. So the harness treats the interviewer as the system under test and grades it on both jobs.

The harness

One eval run takes a synthetic candidate with a known answer key, puts it in front of the real interviewer, and hands the resulting transcript to two panels of judges. Everything below is a detail of this picture.

Synthetic candidates with a known answer key

Evaluating an interviewer is hard because you normally don't know how good the candidate really was, so you can't tell whether the score was right. We get around that by manufacturing the candidate.

The harness defines fifteen synthetic candidates, each with an intended_overall caliber on a 1–10 scale baked in as ground truth. They span the three interview types, seniority from L4 through L6, and three strengths: weak, mid, and strong. A mock candidate model role-plays each persona and stays in character, so a weak L4 gives shallow, hand-wavy answers while a strong L6 reasons crisply about trade-offs. Because we know what each candidate was supposed to demonstrate, we can ask whether the interviewer's score landed where it should have.

The interview loop

The runner drives the production Interviewer class against the MockCandidate on the same code path real users hit, so what gets evaluated is what ships. The loop alternates until the interviewer emits a "done" sentinel, or until it hits --max-turns as a safety cap.

Letting the interviewer decide when it's done is itself an eval signal, because an interviewer that bails out after two turns regardless of who's in front of it is failing a different kind of probing test. That turns out to be the most revealing thing the harness measures, and it gets its own companion piece.

Two judging jobs, three model families

Each interview is graded twice over, by two different rubrics, and each rubric is scored independently by three models from three labs:

• The experience judge reads the transcript and rates the interviewer on six dimensions: question_relevance, probing_depth, adaptivity, did_not_leak_answers, professionalism, and coverage.
• The feedback judge reads the transcript and the generated feedback and rates five dimensions of the write-up: specificity, actionability, grounded_in_transcript, fairness, and internal_consistency.

The feedback judge is blind to the persona's intended caliber. Before it sees the interviewer's score it forms its own estimate of how the candidate performed, recorded as demonstrated_caliber, and that one design choice is what makes the calibration breakdown below possible.

Splitting "the score was wrong" into whose fault it was

The naive calibration metric asks how far the interviewer's score sits from the intended caliber. That conflates two failures: the interviewer mis-scoring a good performance, or the mock candidate having an off run and not delivering its intended caliber. We separate them into three numbers:

• calibration_error = │interviewer's score − intended caliber│. The headline number, though it mixes both failures.
• interviewer_calibration_error = │interviewer's score − blind judge's caliber│. The interviewer's own scoring accuracy, independent of whether the candidate drifted.
• candidate_fidelity_error = │blind judge's caliber − intended│. How far the mock candidate drifted from its brief, which is a property of the test rig rather than the interviewer.

A low interviewer_calibration_error next to a high calibration_error means the interviewer scored the transcript in front of it correctly, and the mock candidate just didn't perform to spec that run.

Why the judges come from three different labs

A judge from the same family as the interviewer grades its own house style generously, so every score here averages a cross-family ensemble: claude-opus-4-8, gpt-5.5, and gemini-3.1-pro-preview, at temperature 0 where the API allows it. Two rubrics × three judges means six independent gradings of every interview, and 540 across the ninety interviews in this post. We also record judge disagreement, the mean absolute deviation of their three overall scores. When judges diverge we treat the result as soft rather than averaging it away.

Every run also records a judges_hash (a fingerprint of the judge prompts plus temperature) and a personas_hash. Change a rubric or a persona and the hash changes, so our reporting tool refuses to diff two runs that weren't graded under identical conditions. A blog post full of numbers is only worth reading if the numbers were produced the same way, and the hashes make that checkable instead of a promise.

Running everything twice

Every configuration is run twice, and every number in this post is the mean of the two runs. This turned out to matter far more than we expected. A single interview's probing_depth score barely predicts what the same interview scores on a rerun, as the reliability section shows.

The experiment

We swapped the interviewer model between three Claude tiers and held everything else fixed. Every run in this post was graded under judges_hash 3b4eac88e05a and personas_hash 05e588ff2c02, with the same Haiku mock candidate and the same 20-turn cap. Those hashes, not a commit SHA, are what certify the comparison: a SHA tells you the code matched, the hashes tell you the grading conditions matched, which is the thing that actually has to hold.

Ninety interviews: three models × fifteen personas (five each across behavioral, system design, and ML system design) × two replicates.

The aggregates are macro-averaged across the three interview types (each type weighted equally, five personas apiece), so a type with a wider score spread can't dominate the headline. Where we claim a difference, we report a paired t-test across the fifteen personas, two-sided, α = 0.05.

Results

The tiers land within half a point of each other

There is no cliff on the interview itself, and the quality difference is fractions of a point. Haiku is statistically indistinguishable from both larger models at conducting the interview. Haiku − Sonnet is −0.16 (t = −1.48) and Haiku − Opus is +0.37 (t = +1.44). Our production recommendation rests on that null: the cheapest tier is not measurably worse at the live job.

One experience-quality gap does clear the bar, Sonnet over Opus at +0.53 (t = +2.17), and it clears it by a hair against a threshold of 2.145. We read it as a hint rather than a result. It's the kind of marginal finding a third replicate would be most likely to erase.

The price gap is smaller than the rate card suggests

Opus lists at five times Haiku's per-token price. That is not what you pay per interview, because Haiku is the model that won't stop asking questions. It takes 14.5 interviewer turns to Opus's 8.8, and every turn re-reads the whole transcript, so context cost grows quadratically in turn count. Running the live conversation on Opus costs 1.8× what it costs on Haiku, not 5×. Sonnet is the most expensive of the three to run the conversation, more expensive than Opus, because it talks 42% more for a 40% lower token price.

A note on reading the tables

Comparing the columns by eye will mislead you. Each model's mean carries a wide confidence interval, because the personas differ enormously from one another and a weak L4 and a strong L6 drag any average around. Haiku's probing_depth is 8.70 ± 0.50 and Sonnet's is 8.22 ± 0.46. Those intervals overlap heavily, and a reader would reasonably conclude there's no difference.

There is. Every test in this post is paired. It asks whether one model beat another on the same persona, fifteen times over, which cancels the persona variance entirely. The paired difference on probing_depth is +0.48 ± 0.44 (t = +2.33), significant despite the overlapping marginal intervals. Overlapping error bars on the means never imply a null result for paired data. The inference only runs the other way. So we report confidence intervals on the differences, which is what our claims are about.

Where the differences actually live: the dimension breakdown

The single "experience quality" number hides the most interesting result. Broken into its six dimensions (macro-averaged over 90 interviews):

Orange = significantly ahead of the runner-up; bold^* = numerically best, but the gap to the runner-up is not significant.

Both comparisons against Sonnet clear significance with little room to spare, and their confidence intervals nearly touch zero. The comparisons against Opus are the robust ones. If a reader takes only one claim from this section, it should be that Haiku probes deeper than Opus.

And the feedback-quality dimensions:

Orange = significantly ahead of the runner-up; bold^* = numerically best, but the gap to the runner-up is not significant.

Opus and Sonnet, meanwhile, are indistinguishable from each other on feedback overall (−0.18, t = −1.90). The judges also agree far more about Opus's feedback than Haiku's (disagreement 0.43 vs 1.01), and Haiku's interviewer calibration error is more than double Opus's (0.52 vs 0.21). Haiku probes well but is a less reliable grader of what it found.

The hardest interview for every model: system design

Finding 3: system design is where calibration breaks down, and it breaks the same way for every model, through over-generosity to weak candidates. Calibration error roughly doubles on system design (≈1.2) against behavioral (0.70) or ML system design (≈0.55). The two weak personas show it plainly, and both have an intended caliber of 3.

Every model rounds a weak system-design answer up, and the blind judges saw it too. Their independent demonstrated_caliber for both personas came in at 3.8, so the candidates really were weak and the interviewers over-credited them. These scores are the interviewer's own rating of the candidate, which reproduces across reruns at r = 0.96, so this table is signal rather than noise. It's the clearest defect the harness surfaced, and the most actionable. Our system-design prompt needs a firmer rubric for what "insufficient" looks like, whichever model we ship.

How much can you trust a single run?

We ran every configuration twice, which let us ask a question most eval write-ups never ask: if you rerun the same persona against the same model, do you get the same score? For the calibration machinery, emphatically yes. For the experience judge, emphatically no.

The interviewer's score of a candidate is almost perfectly reproducible, and tracks the intended caliber at r ≈ 0.92 for all three models. Every calibration claim in this post rests on that.

A single interview's probing_depth score tells you very little about what the same interview would score on a rerun. Between-persona standard deviation is 0.92, and within-persona, run-to-run standard deviation is 0.86. The noise is as large as the signal. Model-level means over fifteen personas are stable, and Opus's aggregate reproduced to within 0.07 across replicates, but individual cells are not interpretable. Neither is a one-run comparison of two models on a single persona.

Split the job

The dimension data points somewhere more interesting than a single winner. Each of the two jobs an interviewer does has a different best model.

• Conducting the interview. Haiku is statistically indistinguishable from both larger models on overall experience quality. On the two dimensions that separate a real interviewer from a polite one, probing and adaptivity, it leads outright and significantly, and it is the cheapest of the three to run.
• Writing the evaluation. Grounding, fairness, consistency, and calibrated scoring all favour Opus and Sonnet decisively. Haiku's write-up is measurably less anchored to what happened.

Limitations & what's next

• The candidate is also an LLM. A Haiku mock candidate is consistent and cheap but isn't a human. candidate_fidelity_error (0.59–0.77) quantifies its drift, and we exclude its share from the interviewer's calibration, but it's still a model grading ground for a model.
• Fifteen personas, not fifteen hundred. Replicates buy confidence in the numbers we have; they do not buy generalization. Whether Haiku still out-probes Opus against candidates unlike our five archetypes is a question this design cannot answer.
• Judges are LLMs. The cross-family ensemble and disagreement tracking guard against any one model's bias, but the ceiling is still "what three frontier models agree a good interview looks like." Haiku's feedback disagreement of 1.01 is a warning that the judges themselves are least sure exactly where we draw our sharpest conclusion.

Next: ship the firmer system-design rubric and re-run to confirm the calibration error drops; prototype the Haiku-conduct / Opus-feedback split as a first-class configuration and eval it end-to-end; and grow the persona roster so the gaps we're calling "small" come with real confidence intervals across a wider candidate distribution.

Where we're taking this: continuous evals

Our offline harness catches regressions before release, but it runs on synthetic personas we wrote ourselves, which is a closed world. Real users are messier, weirder, and more interesting than any persona we'll ever sit down and write. The roadmap from here is about closing that gap.

Pre-merge eval CI

Any PR that touches a prompt file, the Interviewer class, or a feedback module should automatically run a fast subset of the harness, one persona per interview type, and post the deltas as a PR comment. A calibration regression should behave like a test failure. The current friction of remembering to run evals manually is how regressions sneak in.

Drift monitoring, with a noise floor

Even with prompts and judges held constant, model providers ship quiet updates. We want a small "anchor" suite re-run weekly and charted over time. The reliability numbers above set the bar for what counts as a signal: since a single interview's probing_depth reproduces at only r ≈ 0.17, an alert threshold has to sit above the run-to-run noise, not above zero. Any drift monitor that fires on a one-point move in one dimension on one persona will fire constantly and mean nothing.

Cost and latency as first-class eval dimensions

Judging an interview costs two and a half to four times what the interview itself cost to run, since three frontier judges each apply two rubrics. We work that out after the fact. Attaching tokens-in/out and per-turn p50/p95 latency to each persona result would make the quality-versus-cost trade explicit at the moment we read a result. We have no latency measurements at all today, which is why this post makes no latency claim.

Companion piece: why does the smallest model probe the hardest?

The result that shouldn't happen deserves its own treatment. Haiku probes significantly deeper than Opus at every step of the ladder, and a controlled turn-cap experiment shows that turn budget doesn't explain it: given three times the room to press, Opus stops anyway. Read why the smallest model probes the hardest, with the judges' own rationales and the mechanism we think is behind it.

Want to see the interviewer in action? Start a free practice session, or read why the smallest model probes the hardest.