How to Tell Whether an AI Interview Study Is Actually Reliable (A Checklist)

You judge an AI interview accuracy study by what it measures and how honestly it reports its conditions, not by its headline agreement score. A study is reliable when it measures the relevance of what people said rather than how fluent it sounded, discloses its sensitivity to the exact prompt used, judges answers more than once with controls for length bias, compares against a human baseline under identical conditions, and states plainly where it failed. Most vendor studies do none of these five things, which is why a “95% agreement with human moderators” claim tells you almost nothing on its own.

Here is the number that should reset how you read every one of those claims: trivial changes to a prompt, things as small as extra spaces or reordered examples, can shift an LLM’s task accuracy by up to 76% (Lin, 2025). A study showing 94% agreement might have shown 40% with a slightly different prompt. If the study does not tell you how it handled that, it has not told you anything.

Why this checklist matters now

AI-moderated interviewing has left the novelty phase, which means the burden of proof has shifted from “does it work at all” to “can you show the data holds”. Gartner’s 2025 Hype Cycle moved conversational AI for research past peak hype and into the stage where buyers can make sound choices. The tooling is real, and the vendor field is crowded with accuracy studies of wildly uneven quality. An interview can be efficient and still unreliable, structured and still shallow. Efficiency is what these tools are sold on. Reliability is what you need. They are not the same test.

The 2025 and 2026 research gives you real tools to tell the difference. Five tests, each grounded in a named, checkable source.

Test 1: Does it measure relevance, or just richness?

Ask whether the study scores whether answers were on-point, or whether they merely sounded detailed and articulate. This is the test most studies fail, and the finding behind it is recent enough that most of the industry has not caught up.

In 2026, researchers at Johns Hopkins ran the largest validation study of interview quality metrics to date: 343 real interview transcripts containing nearly 17,000 participant responses across 14 actual research projects (Ivey, Field and Xiao, 2026). They took ten commonly proposed measures of response quality and tested which ones predicted whether a response actually contributed to the study’s findings. The result: relevance to a key research question was the strongest predictor. Two of the metrics the industry leans on hardest, clarity and surprisal-based informativeness, were not predictive at all.

Read that against the average vendor benchmark. Most report that their AI elicits “longer”, “richer”, or “more detailed” responses. Easy to measure, impressive on a chart, and per the best evidence we have, not quality. A respondent on autopilot can produce three articulate paragraphs that answer nothing the study needed answered.

What to look for: a study that defines quality as contribution to the research question. If every reported metric is about how answers sounded rather than what they did, the study measured the surface and skipped the substance.

Test 2: Was prompt sensitivity reported?

A study that tested one prompt configuration, once, has told you how that exact setup behaved on that exact day, not how the system behaves. Back to the 76% problem: LLM outputs are unstable under superficial prompt variation, and models can even agree with contradictory statements depending on framing (Lin, 2025). A single-prompt evaluation is a point estimate drawn from a distribution nobody explored.

Reliable studies handle this in one of two ways. They run multiple prompt variants and report the spread, or they standardise conditions so tightly that variation is controlled by design. A 2026 evaluation in Nature Scientific Reports did the latter: to compare AI interviewers fairly, the team standardised context using transcripts from ten baseline human interviews and executed every model under identical orchestration and prompts (Panfilova et al., 2026). That is what methodological care looks like.

What to look for: prompt variants, sensitivity analysis, or standardised orchestration. If the methods section is silent on how the prompt was chosen, the absence of a range is itself a finding. It usually means nobody looked, or somebody looked and preferred not to publish it.

Test 3: Who judged the answers, and how many times?

If an AI model graded the AI interviewer’s output, and ran that grading once, the study has two unexamined weaknesses stacked on top of each other. LLM judges systematically prefer longer answers regardless of whether the extra words add anything, and the same judge scoring the same answer twice can disagree with itself (Kim et al., 2025). A single pass by a biased judge is not a measurement. It is one opinion, delivered once.

And agreement is not validity. Combine Test 1 and Test 3 and you can see how a weak study manufactures a strong result: an AI that produces longer responses, scored by a judge that prefers longer responses, using a metric (informativeness) that does not predict quality. Every step is defensible in isolation. The result is meaningless.

What to look for: human evaluation, or at minimum multiple judge runs with reported agreement, and an explicit control for length effects in scoring.

Test 4: Is there a human baseline under identical conditions?

An accuracy claim with no human comparison run under the same conditions is a number without a denominator. “Our AI scored 4.2 out of 5 on engagement” tells you nothing without knowing what a human moderator scores with the same questionnaire, population and measures.

The studies worth trusting run true comparisons. One frequently cited example randomly assigned participants to either an AI or a human interviewer using identical questionnaires, then assessed guideline adherence, response quality and engagement across both arms (Wuttke et al., 2025). The honest picture is nuanced: AI interviewers hold structure and consistency extremely well, probe reasonably, and occasionally miss the emotional beats a skilled human catches. Nuance is what real findings look like. That design is expensive and inconvenient, which is exactly why its presence is a quality signal and its absence is a warning.

Results also do not automatically transfer across markets and languages. A baseline built on US English respondents proves less than it appears to elsewhere. A serious study states its scope rather than implying universal reach.

What to look for: random assignment, identical instruments in both arms, and results that include things the AI did worse. A comparison where the AI wins every dimension is a comparison designed for the AI to win.

Test 5: Does it report failures, uncertainty, and limitations?

The fastest test of the five, and the most reliable tell: serious research reports what went wrong. Sample sizes are stated. Preprint status is acknowledged (much of this field is on arXiv and not yet fully peer reviewed, including some papers cited here, which is exactly what an honest write-up says out loud). Confidence is qualified.

Marketing dressed as research reports a clean sweep: every metric up, every comparison won, no limitations section. The absence of reported failure is not a sign of a strong product. It is a sign of a filtered study. If the failures are missing, they were not absent. They were edited out.

What to look for: a limitations section that actually limits something. If the authors will not tell you where their system struggles, they are not describing a system. They are describing a pitch.

The transcript test

No published study, however clean, substitutes for reading transcripts from your own pilot on your own population. The five tests tell you whether to take a study seriously. Only your own data tells you whether the platform works for your questions, your people, your language.

The test is boring and it works. Run a small campaign. Read the transcripts, not the dashboard. For each one, ask a single question drawn from the Johns Hopkins finding: did this conversation produce responses relevant to what the study needed to learn? Did the moderator probe when an answer was vague? Did it stop once it had enough? You will know within a dozen transcripts.

If you are designing that pilot from scratch, we wrote a full companion guide: how to design a reliable AI-moderated interview study. The tests in this piece are the audit; that guide is the build.

And if the platform cannot hand you raw transcripts to read, that is your answer. The reluctance is the finding.

Where Tadeus stands

One thing transcripts alone cannot show you: two sessions can produce near-identical text where one respondent was reflective and the other was in completion mode, answering to be done rather than to be understood. Today every Tadeus session ships with structured per-session metrics scored from the transcript and the audio signal itself, including relevance to the campaign’s research objective, the one metric the current evidence says actually predicts quality, alongside voice-level comfort signals and a six-dimension emotional profile (where permitted under the EU AI Act and similar regulation). The next layer, in development now, is a behavioural quality score derived from response latency, word count compression across turns, and silence patterns, designed to flag completion-mode sessions to the researcher rather than quietly hide them. Some of our sessions will carry flags that make the data look less confident than a competitor’s output would. Per Test 5, that is a feature.

We are not going to hand you a single agreement number and ask you to trust it. That number would fail Test 3 and Test 4 on its own terms, and we wrote the tests.

The checklist

When someone hands you an AI interview accuracy study, ask:

1. Is quality defined as relevance to the research questions, or as length, fluency and richness?
2. Did they test prompt variants, or run everything once on one prompt?
3. Who scored the outputs, and were length bias and run-to-run consistency controlled?
4. Is there a randomised human baseline under identical conditions?
5. Is there a limitations section that names something real?
6. Can you run a pilot and read the transcripts yourself?

A study that passes four or more is worth your attention. A study that fails the first two is worth your scepticism regardless of the headline number.

The AI-moderated interview category no longer needs to be oversold. The methods work, the research is maturing, and the honest platforms can afford to invite scrutiny. Run the pilot. Read the transcripts. The evidence will tell you what the benchmark charts will not.

Want to run that pilot? Set up a Tadeus campaign, interview ten people, and read every transcript with the checklist beside you. That is the evaluation we would want you to run on anyone, including us.

Frequently asked questions

What does “reliable” actually mean for AI-moderated interviews?

Not that the interview repeats identically. It means the data reflects what respondents genuinely think and addresses the questions asked. Per the strongest current evidence (Ivey, Field and Xiao, 2026), relevance to the research question is the metric that predicts quality; fluency and informativeness do not.

Is a “95% agreement with human moderators” claim good evidence?

On its own, no. Agreement is not validity: you can reach high agreement with a rubric that rewards the wrong signals, and LLM judges are known to prefer longer answers and disagree with themselves across runs (Kim et al., 2025). Ask what the number was measured against, whether a human control group answered the same questions, and whether the grading was repeated.

Why shouldn’t I trust fluency or richness scores?

Because the largest empirical study of interview response quality found that clarity and informativeness do not predict whether a response contributes to research findings, while relevance does (Ivey et al., 2026, 343 transcripts, roughly 17,000 responses). A fluent, detailed answer can be completely irrelevant, and an autopilot response often reads as richly as a considered one.

How big does a pilot need to be before I trust my own read?

Small. Reading a dozen or two full transcripts from your own population, judging each on whether it answered the question that was asked, will usually reveal the pattern. Scale up once the small read gives you confidence, not before.

Our design guide for AI-moderated interview studies covers how to set the pilot up.

Does the language or market change how I should evaluate a study?

Yes. Results do not automatically transfer across languages and populations, so the human baseline and prompt-sensitivity evidence need to exist in the market you operate in. Randomised AI versus human interviewer comparisons have been run on German-speaking samples (Wuttke et al., 2025), which is the kind of scope-specific evidence to ask for.

References

• Ivey, J., Field, A., Xiao, Z. (2026). What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews. arXiv:2604.05163
• Lin, Z. (2025). From Prompts to Constructs: A Dual-Validity Framework for LLM Research in Psychology. arXiv:2506.16697
• Kim, E. et al. (2025). LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation. arXiv:2412.10424
• Wuttke, A. et al. (2025). AI Conversational Interviewing: Transforming Surveys with LLMs as Adaptive Interviewers. arXiv:2410.01824
• Panfilova, A. et al. (2026). The AI Interviewer: Multi-Faceted Evaluation of Adaptive Questioning by Large Language Models. Scientific Reports
• Jacobsen, R.M. et al. (2025). Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes. CHI 2025
• Gartner (2025). Hype Cycle for Artificial Intelligence