8 AI Hiring Mistakes Recruiters Make and What to Do Instead
Practical fixes for the places AI goes wrong in screening, ranking, drafting, disclosure and record keeping

AI hiring mistakes can come from features you switched on for good reasons, such as faster screening or a quicker first draft. Your ATS (applicant tracking system) may now screen, rank and write to applicants, and other tools may handle sourcing or scheduling. Each feature makes choices about candidates, so it helps to know where those choices can drift from what you intended.
1. Letting software reject on judgment calls
Keep automatic rejection for objective must-haves, such as a required license, and send the rest to a recruiter. Judgment calls depend on context a filter can't read. Knockout questions help with volume, but trouble starts when a preference becomes a hard requirement.
A required-degree knockout, for example, removes an applicant with years of equivalent experience before anyone reads the resume. Years-of-experience cutoffs work the same way, counting time rather than what someone did with it.
A review queue keeps most of the speed, with the tool flagging applicants who miss a preference and a recruiter deciding whether the gap matters. When you compare candidate screening software, check whether it can flag applicants rather than only reject them.
Work authorization can be a must-have where it's lawful to ask, so confirm that for each location first. For agency recruiters, the client draws the line between requirement and preference, so intake is the moment to agree on it.
- What to do instead: List your current knockout questions and move any that rely on judgment, such as degree or experience, to a review queue.
2. Trusting a match score without the reasoning
Ask for the reasons behind each ranking before you act on it, because you can check a score you can explain against the job. Match scores help order a long list, but the order also influences the person reading it.
In a 2025 lab study, University of Washington researchers gave 528 online U.S. participants simulated AI recommendations as they chose candidates. Without suggestions, their choices showed little bias, but with recommendations they mirrored the AI. When the recommendations were strongly biased, participants' choices followed the AI's picks about 90% of the time.
The participants weren't recruiters at work, and the AI was simulated, but the study shows how a ranked list can become the answer rather than a starting point. Reading a few profiles before you open the scores gives you your own view for the tool to confirm or challenge. When the two disagree, the score explanation shows whether the tool spotted something you missed or weighted something the job doesn't need. If you're evaluating AI resume screening tools, ask vendors to show a plain-language reason for each ranking.
- What to do instead: Review a handful of profiles before you look at match scores, then compare your shortlist with the tool's.
3. Skipping a test on roles you know
Test a tool on a role you've recently filled before you rely on it for new ones, because you already know which candidates were strong. A feature that looks right in a vendor demo may behave differently on your applicant pool.
A filled role lets you catch AI hiring mistakes on candidates you've already assessed. Compare the tool's ranking with the people you advanced and hired. If strong candidates land near the bottom, find out which criteria pushed them down, such as a job title worded differently from the posting. Confirm with whoever owns your candidate data that reusing a past pool this way is allowed.
Agreement with past decisions shows the tool matches your team's judgment, not that either one is fair, so this check complements the bias review below. One role won't tell you enough, since a tool can rank well for engineering roles and poorly for sales roles. Repeat the check across your main job families and after any vendor release that changes screening or ranking. Agency recruiters can repeat it per client.
- What to do instead: Pick one filled role from each of your main job families and compare the tool's top picks with the candidates you advanced.
4. Assuming the tool is neutral
Check screening and ranking tools for bias on a schedule, because AI hiring bias can enter through training data, your criteria, or stand-ins for protected traits. Software applies the same rules to every applicant, so its output can look fair even when those rules favor some groups.
A University of Washington team tested three open-source language models on more than 550 real resumes with the names swapped. In that 2024 study, the models favored white-associated names 85% of the time and Black-associated names 9% of the time. These were research models, not commercial hiring tools.
The method is still one you can borrow. With the vendor's agreement and outside live roles, submit resume pairs that differ only by name and compare where they rank.
Proxies can do the same work without naming a protected trait. Illinois law now bars employers there from using zip codes as a proxy for protected classes. Graduation year can track age similarly. Knowing how AI hiring works helps you trace a skewed result back to its input.
A formal audit takes time and expertise. The Society for Human Resource Management (SHRM) found in 2024 research that organizations not using AI cited a lack of resources to audit AI algorithms. Sampling is a lighter place to start, because the lowest-ranked applicants show whether qualified people are collecting there and why.
- What to do instead: On a set schedule, read a sample of the lowest-ranked applicants for one open role and note anyone you would have advanced.
5. Sending AI drafts without reading them
Read every AI-drafted job description and candidate message before it goes out, because a draft can add requirements or details that aren't true for the role. Among in-house human resources (HR) professionals using AI for recruiting, SHRM's 2024 research found 65% used it to help generate job descriptions.
Generated drafts read smoothly, which makes errors harder to spot. A job description might gain a degree requirement or a skill the hiring manager never asked for. Outreach can name the wrong location, pay range, or start date when the tool fills a gap with plausible wording.
An added requirement shapes who applies, and it can slip into the knockout questions from the first item. Checking pay and location against the approved job details catches factual slips. Agency outreach represents both the client and your firm, so agree on message templates with each client before using them.
- What to do instead: Keep the hiring manager's intake notes open while you edit each AI draft, and remove any requirement that isn't in them.
6. Leaving candidates and clients uninformed
Tell candidates where AI is part of your process and how to reach a person or ask for an accommodation. A notice lets them flag AI hiring mistakes you can't see from your side, such as a misread resume. Disclosure can feel optional when the tool sits inside your ATS, but some places require notice.
New York City's Local Law 144 applies to employers and employment agencies using automated employment decision tools for NYC jobs. Under that law, a covered tool needs a bias audit within one year of use, public audit information, and notices to candidates or employees. Illinois Public Act 103-0804 requires Illinois employers to notify employees when they use AI, effective January 1, 2026. Its notice language refers to employees, so ask counsel how it applies to applicants.
SHRM's 2026 research found 57% of HR professionals in states with workforce-related AI regulations weren't aware of those rules. Requirements depend on where each role is located, so check them with employment counsel.
Brief clients and hiring managers before launch too, since they'll field questions about how candidates were screened. If you use a recruiting chatbot, its first message is a natural place for a short notice and a way to reach a recruiter.
- What to do instead: Add one sentence to job postings and candidate messages that names where AI is used and how to reach a recruiter instead.
7. Not measuring what changed
Record a baseline before you switch on an AI feature, because without one you can't tell whether it helped, hurt, or changed nothing. Where the filled-role test checks whom the tool picks, a baseline covers the whole process.
A 2026 SHRM report found that 56% of HR functions don't formally measure the success of their AI investments. The same research found only 16% use ROI (return on investment) as a metric.
When features go live mid-search, the before picture never gets captured. Candidate drop-off at each stage, time to first recruiter review, and hiring manager feedback on shortlists together cover speed and quality. Speed alone can mislead, since a tool that rejects more applicants makes every later stage look faster.
Capture those numbers before launch and again afterward. An HR dashboard can track them if your ATS reports by stage. If a number moves, check whether something else changed too, such as a new job board or a hiring freeze.
- What to do instead: Export your stage-by-stage numbers for current open roles before the next AI feature goes live.
8. Keeping no record of AI decisions
Keep a simple log of which AI tools and settings screened each role, who approved them, and when they changed, so you can answer questions later. Settings shift through vendor updates, new knockout questions, or a colleague's adjustment. With a log, you can trace AI hiring mistakes back to the setting that caused them.
Retention rules apply to hiring records too. Under federal recordkeeping rules, the U.S. Equal Employment Opportunity Commission requires covered private employers to keep application forms and hiring records for one year. In California, Civil Rights Council rules effective October 1, 2025, require employers to keep employment records, including automated-decision data, for at least four years.
Ask counsel which rules cover your roles and which AI outputs, such as scores, to keep. Agencies may find a separate log for each client simpler to search and to share when a client asks. A log matters more as agentic AI tools take on multistep tasks, since fewer steps pass in front of a recruiter.
- What to do instead: Start a shared log with columns for role, tool, settings, approver, and date changed, and add an entry whenever a setting moves.
Check for AI hiring mistakes before you add more tools
Start with an inventory of what's already running. Open your ATS and any connected tools, list the AI features that are switched on, and note what each one decides or drafts. Then compare that list with what your team and clients believe is happening, because the gaps are where AI hiring mistakes hide. That inventory gives you a clearer basis for deciding where AI recruiting features belong next.
See why each candidate matches before you decide
Book a demoFrequently asked questions
Is it legal to use AI in hiring?
The answer depends on where each role is located and how the tool is used. Some places, including New York City, Illinois and California, have rules on notices, bias audits or records. Check the rules for each role's location with employment counsel.
Who is responsible when an AI hiring tool gets something wrong?
It depends on the law and your vendor contract, so it's a question for counsel. Some local rules, such as New York City's, place duties on the employers and employment agencies using a tool. Review vendor contracts to see who handles audits, notices and records.
What is AI hiring bias?
AI hiring bias is when a screening, ranking or matching tool disadvantages applicants based on protected traits, such as race, gender or age. It can happen without anyone intending it, because a model can learn patterns from its data or from stand-in fields. Comparing outcomes across groups is one way to spot it.
About the author

Founder & CEO of Rotto, building tools that help tech recruiters source better candidates, faster.





