AI Candidate Evaluation: How to Use AI Scores and Summaries Without Handing Over the Decision
What AI scores, ratings and interview summaries can tell you, what to check before you rely on them, and how to keep people responsible for hiring decisions

For many recruiters, AI candidate evaluation lives inside tools they already use, not as a separate purchase. An assessment platform returns a score, a video interview tool rates recorded answers, and a meeting app drafts a summary before you've written your own notes.
AI candidate evaluation uses AI to score, rate, summarize, or compare candidates already in a hiring process based on their tests and interviews. It saves reading and note-writing, but each output is only as reliable as the evidence behind it. That's why the hiring decision stays with people who have checked that evidence.
Key takeaways
- Before relying on an AI score, find out which inputs produce it and whether they relate to the job's requirements.
- Compare AI summaries with the transcript or your own notes, since a summary can drop the context behind an answer.
- Tell candidates when AI analyzes their interview, and offer another way to be assessed if they need one.
- Have a named person review any decision that moves a candidate out, and record who decided and why.
What is AI candidate evaluation?
The clearest way to separate it from screening is timing: screening decides who enters your process, and evaluation assesses the people already in it. Screening sorts a large applicant pool with knockout questions and résumé ranking, which automated candidate screening handles. Evaluation comes later, when the list is shorter, and each score or summary feeds a decision about one person.
That shift changes what you're checking. With AI resume screening, the main question is who gets filtered out of a long list. After the screen, the question becomes whether a score or summary fairly represents one person, since a hiring manager or client may read it instead of the interview.
These tools are far from standard in HR. In a 2026 SHRM survey of 1,908 HR professionals, 39% of organizations had implemented AI in HR functions, and 27% had implemented it in recruiting. A 2024 SHRM survey found that about 33% of HR professionals using AI for recruiting used it to review or screen résumés. Both surveys describe in-house HR teams, not agencies.
Where AI evaluates candidates after the first screen
After the first screen, AI shows up in four main places: scored assessments, analyzed video interviews, transcript summaries, and drafted scorecards or debrief notes. Some features come built into an applicant tracking system (ATS), and others come with dedicated candidate evaluation software. The table shows what each produces and what to check first.
| Where AI shows up | What it produces | What to check before you use it |
|---|---|---|
| Skills assessments and work samples | A score or a ranking against other test takers | Whether each scored item relates to the job's requirements |
| Video interviews analyzed by AI | Ratings or rankings of recorded answers | Which characteristics the tool rates and whether candidates consented |
| Interview transcripts and summaries | A written account of what the candidate said | Whether the summary matches the transcript and keeps the context |
| Scorecards and debrief notes | Draft ratings and comments for the hiring team | Whether each rating points to something the candidate said or did |
Scored assessments are the most familiar option, and a work sample score can help when the task resembles the job. What matters is what the score rewards, whether that's speed, accuracy, specific wording, or something closer to the work. When you're choosing pre-employment assessments, ask the vendor for that breakdown before the first candidate takes the test.
AI-analyzed video interviews require more care because candidates can't see what's being rated, and the format doesn't suit everyone. When comparing video interview software, it helps to separate tools that record and transcribe from tools that analyze the answers. AI analysis of a recording also triggers notice rules in some places, including Illinois.
Summaries and drafted scorecards save the most time, and people trust them readily because they read like a colleague's notes. A summary keeps what the model judged important, and it can drop the hesitation, follow-up question, or qualifier that explained an answer. Agency recruiters have an extra reason to check them, since a client may receive an AI-drafted write-up under your name.
What does an AI candidate score measure?
A score reflects whatever inputs and weightings the vendor built into it, which may match your role closely or only loosely. Connecting each rated item to the job requirements tells you which ones matter. If your interview rubric already defines a strong answer for each requirement, compare the tool's rating criteria with those definitions.
Most of what you need comes from the vendor, whether you're choosing a tool or reviewing one that came with your system:
- Inputs: Which parts of the test, recording, or transcript feed the score, and which are ignored?
- Explanation: Can you see the factors behind one candidate's result, not only the overall number?
- Evidence links: Does each rating or summary point back to the answer or passage it came from?
- Control: Can you switch off AI analysis for a role or give a candidate a format without it?
- Records: Can you export scores, inputs, and reviewer decisions for audits or record retention?
These questions carry more weight for AI candidate scoring that ranks people against each other, because a small weighting choice can reorder a shortlist. A number you can't explain to a hiring manager, client, or candidate is safer treated as a reason to look closer than as a result.
Model ratings can also shift in hard-to-predict ways. In Wharton research testing 11 large language models (LLMs), the team changed only names and pronouns in résumés and interview transcripts. The models rated women and people of color slightly more favorably than White men, but the differences were modest, and the direction wasn't always predictable.
The researchers concluded that organizations need context-specific audits. For a recruiting team, that can mean running past candidates whose outcomes you know through the tool and comparing its ratings with what your interviewers concluded. When the two disagree, it's often worth understanding why before the tool sees live candidates.
Check the evidence behind each rating before you act on it
Before acting on a rating or summary, find the answer, work sample or transcript passage that supports it. A rating that points to something the candidate said takes a minute to check, while a general label gives you nothing to verify.
Checking the evidence also counters a tendency researchers have measured. In a University of Washington study of 528 online U.S. participants given simulated AI recommendations, choices showed little bias without suggestions but mirrored the AI with them. When the AI was severely biased, choices followed its picks around 90% of the time.
That was a lab study rather than recruiters at work, but it echoes a concern raised in a SHRM Advisor blog. The opinion piece argued that with algorithms, decision-making has become "a matter of score acceptance rather than evaluation" and called for people to keep decision authority.
You don't need to recheck every line. Compare the summary with the transcript or your interview notes on the points that will drive the decision, such as the strongest and weakest answers and flagged concerns. Where they disagree, go with the source and note the discrepancy, since a summary error may tell you more about the tool than the candidate.
Check why each candidate matches before you reach out
Book a demoCandidate notice, consent and alternative formats
Telling candidates when AI evaluates them, and offering another format when someone needs it, is good practice anywhere and a legal requirement in some places. Illinois has had rules for AI-analyzed video interviews in force for several years.
Under Illinois's video interview law, an employer using AI to analyze recorded interviews for Illinois-based positions must notify applicants beforehand. It must also explain how the AI works, what general characteristics it evaluates, and obtain consent. The law limits who can view the videos and requires employers to delete them within 30 days of an applicant's request.
Employers that rely solely on AI video analysis to choose who gets an in-person interview must also report demographic data. A July 2025 state report said none was reported for the 12 months ending November 30, 2024. In New York City, Local Law 144 requires a bias audit within one year of use and candidate notices for covered automated tools used for NYC jobs.
On Illinois-based or NYC roles, agency recruiters can agree in writing with the client on who sends notices and collects consent. The rules depend on the role's location, so check with counsel when a search touches a regulated location.
Accessibility belongs alongside notice. A 2026 SHRM article on Americans with Disabilities Act (ADA) compliance shared one speaker's checklist for AI tools. It asks whether the tool works with assistive technologies and whether video interviews have been mapped as an ADA risk area. It also asks how accommodations are triggered before negative actions.
The checklist isn't a legal standard, but its questions work well with vendors. Practitioner advice in SHRM's video coverage recommends not making video the only option and telling candidates they can interview by phone or in the office.
Human review for decisions that move candidates out
Keeping a named person on any decision that rejects a candidate means someone has looked at the evidence before the candidate hears the outcome. The same SHRM checklist asks whether a process ensures human review before finalizing adverse decisions. It also asks whether there's a documented way to appeal AI decisions, such as a contact who can request a second review.
For agency recruiters, the client makes the final hire, but you decide who to present, so the same review applies to anyone you drop from a shortlist. Given how readily participants mirrored AI picks in the University of Washington study, order matters too. In the interview debrief, interviewers can share their ratings before seeing the tool's view, so the AI output becomes one input rather than the starting point.
Record who made each decision and why, in the ATS or wherever your team keeps hiring records. California's rules, effective October 1, 2025, say an automated-decision system may violate state law if it harms applicants or employees. They also require employers to keep employment records, including automated-decision data, for at least four years.
At the federal level, Equal Employment Opportunity Commission (EEOC) recordkeeping rules require private employers to keep application forms and hiring records for one year. Retention can be longer in some situations, so confirm with counsel.
Measuring whether AI ratings match your hiring outcomes
Comparing the tool's ratings with what happened next is the clearest test of whether AI candidate evaluation is helping your team. The same 2026 SHRM survey found that more than half of HR functions don't formally measure the success of their AI investments. These comparisons show whether the tool agrees with what your team saw:
- Agreement with interviewers: Compare the tool's ratings with interviewers' independent ratings for the same candidates. Consistent gaps suggest you should look at what each measures.
- Later-stage results: Check whether candidates the tool rated highly move through later stages and receive offers at a different rate from those it rated lower.
- Pass-through by group: Where you have demographic data collected voluntarily and lawfully, compare pass-through rates by group at each AI-assisted step.
- Vendor updates: Recheck after the vendor changes its model, since the Wharton researchers noted a model may behave differently depending on task, prompt, or population.
Include the candidate side too, such as drop-off rates for AI-scored assessments, one-way video, and how often candidates ask for another format. For agency teams, these records also help when a client asks how you built a shortlist.
Start with one evaluation step before you widen AI use
These checks are far more manageable for one stage than for the whole process at once. Transcript summaries or a scored assessment are sensible places to begin, since AI may already touch your candidates there. Once that stage's outputs hold up against your notes and outcomes over a set of hires, extend the same checks to the next one.
Start with a search you can check before you shortlist
Book a demoFrequently asked questions
Should candidates see their AI score?
Sharing a score is a policy choice for your team and the employer. What candidates need, and in some places are legally owed, is a clear explanation of how AI is used and a way to ask for review. If you do share scores, explain what each one measures so it isn't read as a verdict on the person.
Can AI candidate assessment judge soft skills on video?
Some tools rate recorded answers for qualities such as communication, so ask the vendor what drives the rating. Treat the rating as a reason to review the recording, since on-camera comfort varies for reasons unrelated to the job, such as disability or nerves.
Is AI candidate evaluation legal?
Whether a particular use of AI candidate evaluation is allowed depends on where the role is and how the tool is used, so check with counsel. Illinois, New York City and California all have rules covering notice, audits or recordkeeping. The 2026 SHRM survey found more than half of HR professionals in states with workforce-related AI rules weren't aware of them.
About the author

Founder & CEO of Rotto, building tools that help tech recruiters source better candidates, faster.





