The company Amazon has published an independent validation of AI-conducted interviews in the Amazon Connect Talent tool
The company Amazon has published an independent assessment of the accuracy of AI-conducted interviews in the Amazon Connect Talent tool (launched in September 2026). According to Amazon, the AI's assessment agreement with experts reached 89.3%, 97.6% for clear evidence, and Cohen's kappa was 0.79.
The company Amazon has published details of an independent assessment of the accuracy of AI-conducted interviews in the Amazon Connect Talent tool, which it launched in September 2026 for mass hiring. According to Amazon, an independent firm specializing in industrial and organizational (I-O) psychology reviewed 36 interviews and 252 evaluation signals and compared them with the system's own assessment. The agreement between the AI's assessment and the experts reached 89.3% of cases, 97.6% for answers with clear evidence, and 92.9% for signals indicating problematic behavior. The statistical agreement measure Cohen's kappa was 0.79, just below the 0.81 threshold referred to as "almost perfect agreement."
According to Amazon, the same independent firm also assessed the quality of AI-generated interview content before human editing: in 100% of the assessed position descriptions, the system preserved key information, 91% of generated competencies were clearly derivable from the position description, 98% of questions targeted the intended competency, and 94% of evaluation signals clearly measured the given competency. Amazon states that in the system, the interview and the assessment are separate processes – the AI conducting the interview does not award a score, that only happens afterward based on the transcript, and missing evidence leads to the competency being marked as unassessed, not automatically to a worse score.
The Amazon Connect Talent tool was originally introduced as an AI solution for companies with mass hiring (retail, logistics, hospitality). Recruiters configure criteria and questions in it according to the job position, AI agents then conduct interviews and candidate assessments available 24 hours a day, and recruiters subsequently receive a dashboard with scored candidates, transcripts, and the reasoning behind the assessment for the final decision. According to Amazon, the assessment is text-based, without influence from the candidate's appearance or voice and without identifying data, and is based on the principles of I-O psychology and structured interviews.
Why it matters
For companies with high-volume hiring, this is evidence that AI assessment of candidates in this specific tool achieves, according to the manufacturer, a measurable level of agreement with human experts, which can strengthen the defensibility of the process in terms of fairness and compliance when scaling hiring. However, Cohen's kappa of 0.79 is just below the "almost perfect agreement" threshold, so this represents a documented, not absolute, agreement with human assessment. For job applicants, the practical consequence is that the system assesses interview answers purely based on the content of the transcript, and actively seeks out missing details through follow-up questions.
What was added since the original report
Verified updates
-
Interviews are structured around specific competencies required for the role; The system uses adaptive follow-ups allowing candidates to better explain their experience; Scoring is separated from the interview and is not influenced by the candidate's appearance or voice; Scoring is text-based and free of bias based on the candidate's identity; The approach is based on the scientific principles of I-O psychology and structured interviews
- Interviews are structured around specific competencies required for the role
- The system uses adaptive follow-ups allowing candidates to better explain their experience
- Scoring is separated from the interview and is not influenced by the candidate's appearance or voice
- Scoring is text-based and free of bias based on the candidate's identity
- The approach is based on the scientific principles of I-O psychology and structured interviews
Two audiences, two different impacts
What this means
For individuals
Candidates who go through an interview via Amazon Connect Talent are assessed only based on the text transcript of their answers, without any influence from appearance or voice, and the system adaptively asks further questions if an answer does not contain enough specific details.
For a business
Companies with high-volume hiring (retail, logistics, hospitality) have access to a tool whose assessment agreement with independent experts was, according to Amazon, measured at 89.3% (up to 97.6% for clear evidence), providing evidence to defend the fairness of the process in scaled hiring.
People and managementCheck the original
Event sources
clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.